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FOREWORD 


This report is prepared In two volumes. Volume I reports 
the findings pf the research. Volume II contains the Appendices 
to the final report. The appendices contain detailed documentation 
of the tools used to conduct the research. This Includes a sample 
set of displays presented to subjects during computer aided testing, 
a set of eifperimenter Instructions necessary to operate and modify 
the programs and a table of contents on the 1981 Symposium on 
Aviation Psychology supported by this grant. 
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EXECUTIVE SUMMARY 


A critical In-fllght event (CIFE) Is a situation which is 
unexpectedit unplanned, unanticipated and is perceived by the pilot 
in conunand to threaten the safety of the aircraft. The CIFE 
requires pilot judgement beyond routine decision making or pre- 
programmed decision structure. The safety of the aircraft depends 
more on pilot cognitive processes than skilled motor performance. 

This research extends the results of earlier research on pilot 
response to CIFE's by using a computer-aided scenario testing system 
(CAT). The system makes use of an Interactive terminal whereby 
navigation displays, instrument panel displays and assorted textual 
material are presented by computer graphics. Communication between 
subject and computer is accomplished by means of the touch sensitive 
CRT screen. These programs include biographical data, knowledge sur- 
vey, a variety of diagnostic scenarios, a destination-diversion 
scenario, an airport ranking exercise and a combined destination 
diversion/diagnostic test with dynamic state presentation and control 
(PLATO-GAT) . A complete time history of all data inquiries and re- 
sponses is maintained for each individual subject tested. 

Research Objectives 

The objectives of this research were to; 

1) Design and implement a computer aided testing device for 
studying pilot diagnosis and destination-diversion decision 
making. 

2) Develop new scenarios to take advantage of the capabilities 
of PLATO®. 

3) Test a variety of candidate hypotheses concerning the style 
and substance of pilot resource management. 

These objectives grew out of earlier research which focused on 
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full-mission simulation scenarios in a Singer GAT-1 flight trainer 
and simple paper and pencil problem scenarios. The overriding con- 
sideration throughout all this research has been to apply hunan 
factors concepts to pilot Information processing and decision making 
in order to: 

a) ascertain the role of pilot background, experience and 
knowledge in problem diagnosis and decision making; and 

b) describe the problem solving paths in sufficient detail to 
permit the ultimate development of various models of 
pilot behavior. 

Major milestones in the total project development are noted in 
Figure 1. The particular tasks accomplished on the current research 
(CAT) are noted in Figure 2. The Aviation Psychology Symposium^ listed as one 
of those tasks, provided the opportunity to exchange ideas with other 
researchers in the areas of pilot judgement and decision making. 

Computer Aided Testing Formats 

Software for a touch sensitive CRT Computer Graphics Terminal has 
been successfully implemented to perform the biographical question- 
naire, knowledge test and problem scenarios previously accomplished 
by paper and pencil testing. In addition, programs have been written 
which include dynamic navigation and control capabilities used as 
part of a full mission simulation embodying both problem diagnosis 
and destination diversion decisions. A prototype decision support 
system called Airplane Condition Evaluation (ACE) has also been devel- 
oped as a computer aid for pilot decision making in emergency situations. 

The programs are nearly experimenter free. Commiinication between 
subject and computer is accomplished by means of a touch sensitive 
CRT screen. When results of computer aided testing are compared with 
the equivalent paper and pencil data, the most striking difference con- 
cerns the number of inquiries and Information tracks subjects employ. 
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OVERVIEW OF THE RESEARCH 



Figure 2: Overview of the Research 








In most scenarios the paper and pencil subjects make fever 
Inquiries and employ fewer tracks than ae the CAT subjects. The 
impersonal nature of communicating with the computer terminal 
as opposed to the verbal, exchange in earlier paper and pencil 
studies appears to lessen Inhibitions and broaden the alternative 
hypotheses considered by pilot subjects. 

Results from CAT 


As a result of analyzing the Information seeking styles of 
some 40 subjects using computer aided testing^ the following ob- 

I 

servatlons can be made: 

1) More knowledgeable pilots make better diagnosticians 
than less knowledgeable ones. 

2) Pilots have difficulty in identifying the symptoms of 
a vacuum pump failure. 

3) Knowledgeable pilots reach conclusion (right or wrong) 
more rapidly than others. 

4) Less experienced pilots tend to use a larger number of 
diagnostic tracks than do more experienced pilots. 

5) IFR rated airmen receive higher knowledge scores and 
higher diagnostic correctness scores than do VFR rated 
airmen. 

6) High correctness scores are positively related to high 
mean time between inquiries. 

7) Pilots follow a wide variety of different search patterns 
during problem diagnosis. 

8) Individual pilots tend to exhibit similar search strategies 
across different diagnostic scenarios. 

9) Efficient information searching can be recognized in par- 
ticular scenarios and synthesized for general cases. 

10) Problem solving schemata are scenario dependent. 

11) There is no discernible relationship between the way a 
pilot collects information for diagnosis and the way he 
collects information for destination diversion decisions. 


viii 



12) The only discernible learning effect across scenarios is 
Chat time between inquiries i@ reduced with subsequent 
trials. 

13) Recorded per£oi(^ance measures depend more on the content 
of the scenario than on its order position within test 
session. 

14) PLATO-GAT subjects exhibit resource management styles 
similar to chose observed In full mission OAT simulations. 

15) Wlien faced with both problem diagnosis and the need for 
destination diversion decision making In the same scenario, 
pilots show a strong preoccupation with problem symptoms 

at the expense of positional awareness. 

16) Pilots do not keep enroute alternatives in mind In case 
problems do develop. They react to emergencies more than 
they pre-plan for emergencies. 

17) Data most often requested in diversion decisions are celling 
and visibility. Terrain receives a low number of inquiries. 

18) Pilots often neglect available winds aloft inforraation when 
selecting an alternate airport. 


Potential 

The computer aided testing instruments described in this report 
were developed as research tools to be used to better understand the 
decision making styles of pilots faced with critical In-flight events. 
However, based on repeated comments by subject pilots these tools may 
have even greater potential for pilot training. In addition to 
providing a wealth of potential simulated decision experiences, CAT 
could be used to uncover pilot deficiencies in their understanding 
of the nature of CIFE and to help them to develop more efficient search 
habits. This research can help pilots understand their own problem 
solving logic structure and hence enable them to reevaluate their 
approach to diagnosis. 
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Chapcer 1: Introduction 


A. Background 

Research concerning pilot and system response to critical 
in-flight events (CIFE) began in the OSU Department of 
industrial and Systems Engineering with NASA contract NAS 2-10047 
and was extended under NASA Grant NAG 2-75. Results of those 
efforts covering the period of October 27, 1978 to February 28, 
1981 were reported in "An Investigation Into Pilot and System 
Response to Critical In-Flight Events" Final Report, Volume I and 
Volume II, June 1981 (6), 

A critical in-flight event (CIFE) is defined as a situation 
that either develops quickly or over time which is unexpected, 
unplanned, and unanticipated and which is perceived by the pilot 
in command to threaten the safety of the aircraft. The CIFE is 
one which requires pilot judgment beyond routine decision making 
or preprogrammed decision structure. The safety of the aircraft 
depends more on pilot cognitive processes than skilled motor per- 
formance. 

The overall objectives of the early research were to; 

(1) Describe and define the scope of the critical 
in-flight event with emphasis on pilot manage- 
ment of available resources, 

(2) Develop detailed scenarios for both full mission 
and paper and pencil (P/P) testing of pilot response 
to CIFE's. 

(3) Develop statistical relationships among pilot char- 
acteristics and observed responses to CIFE's. 

These objectives grew out of a concern with anomalies in 
reported accidents and incidents in which some pilots or crews 
seemed better able to handle unusual in-flight events than others. 
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Initial efforts began with a concern for the dynamics 
of ClFE's and broad attempts to Identify pertinent research Issues. 
The final products were: (1) a set of scenarios with associated 

hardware and techniques for studying CIFE phenomena In a simple 
flight simulator; (2) a set of paper and pencil scenarios and 
associated techniques for studying pilot diagnostic strategies 
and diversion decision making processes; (3) a set of knowledge 
testing Instruments designed to measure a pilot's understanding 
of aircraft subsystems and troubleshooting; (4) a study relating 
cockpit crew procedural compliance with performance errors. The 
result of these efforts were reported In the earlier study by 
Thomas H. Rockwell and Walter G. Giff In (6) . 

A five-phase model of pilot CIFE response was hypothesized on 
the basis of (a) discussions with experts In Industry and govern- 
ment and (b) observations made about pilot performance In both 
simulator and paper/pencil scenarios. The five phases were: 

(1) Detection 

(2) Diagnosis 

(3) Option generation 

(4) Decision making 

(5) Execution 

Pilot information seeking activities permeated all five phases of 
this process. 

GAT-1 Full Mission Simulation 

All five phases of pilot response were studied in a GAT-1 flight 
trainer using the following three full mission scenarios: 
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(1) Fuel starvation on the active tank (as might be 
encountered because of a loose fuel cap) . 

(2) Partial power failure (as might be caused by a 
broken baffle in a muffler) . 

(3) Naval4 loss (as might be caused by failure of a 
single airborne receiver component). 

A wide range of cockpit management styles and apparent skill 
levels were observed In these simulations. Although it was 
difficult to quantify, "good performance" was easily recognized 
by the observers of the experiment. The elements of "good perform- 
ance" included: 

(1) professional use of the radio 

(2) precise heading and altitude control prior to and 
during the CIFF 

(3) constant awareness of the aircraft position along its 
Intended route 

(4) prompt, but not Instant, response to the onset of the 
CIFE (detection) 

(5) systematic procedure for troubleshooting 

(6) knowledge and use of available ATC resources 

(7) diversion decisions which allowed for further uncartainties . 

In general. It was found that: 

t 

(1) Cockpit management style varies widely among pilots. For 
example, some are extremely self-reliant, others want 
imraediate and extensive help from ATC while still others 
make the decision making process a joint effort with ATC. 

(2) Good stick and rudder pilots seem to have excess 
capability and maintain good stick and rudder perfortf- 
ance during and after the CIFE. More marginal stick 
and rudder pilots, on the other hand, show Increased 
frequency and amplitude of heading and altitude excur- 
sions, and experience communication difficulties in the 
face of a CIFE. 

(3) Pilots who score well on the knowledge test Instruments 
tend to perform well in problem diagnosis and decision 
making. 


Paper and Pencil Experiments 


Paper aiid pencil (P/P) scenarios, and associated experimental 
techniques, were created to streamline the data collection and 
analysis for pilot responses to critical ln-*f light events. Although 
they lacked the high stress environment of the GAT-l experiments, 
these Scenarios did yield useful data on the pilot problem diagnosis 
and decision making strategy phases of pilot response to CIFE's. 

Four separate diagnostic problem situations were presented to 
forty subjects. These scenarios centered about problems presumed 
to be created by: 

(1) an oil leak at the oil-pressure gauge line 

(2) a vacuum pump failure 

(3) a right magneto drive gear failure 

(4) a frozen static port 

The decision making phase of the paper and pen.il experiments 
was an alternator failure during an IFR flight which forced a 
diversion decision on the pilot. 

For the information seeking task required in the diversion 
decision, the pilot was supplied a simplified enroute chart with 
sixteen airports indicated by letter along his flight path. The 
subject was then given two minutes to ask for information about 
any of those airports. For each airport questioned, there were 
six pieces of information the experimenter was prepared to provide: 

(1) bearing and distance from his present location 

(2) celling at the airport 

(3) visibility at the airport 

(4) approach aids available 


(5) ATC services available 

(6) terrain surrounding the airport 

The pilot's Infomnation seeking observations and ultimate 
airport selected were recorded. 

Later, subjects ranked the suitability of these sixteen 
potential diversion airports. The worth, or weights, for the 
variables ATC, weather, time, and approach were obtained by 
regression analysis according to the techniques of conjoint 
measurement. This study was reported In Flathers, 1980 (1) and 
1982 (2) . 

The following observations were made from the paper and 
pencil tests: 

(1) There Is no correlation between knowledge score 
and total flight hours. 

(2) Knowledge score Is correlated with pilot ratings held, 

(3) Pilots good in one section of the knowledge survey 
tend to be good In all sections. 

(4) Diagnosis performance is highly correlated with know- 
ledge scores. 

(5) Knowledge is Inversely related to total number of 
diagnostic inquiries, e.g. , knowledgeable pilots 
reach conclusions (right or wrong) more rapidly than 
others. 

(6) Total diagnostic inquiries is inversely related to 
correctness, this implies that undirected experimenta- 
tion is poor diagnosis style. 

(7) Total diagnosis correctness score is correlated with 
efficiency, i.e., the ability to arrive at a diagnosis 
with a minimum number of inquiries. 

(8) Civil trained pilots place a higher worth on ATC service 
in diversion decisions than do military pilots. 

(9) Private pilots place a higher worth on weather factors 
in diversion decisions than do commercial and ATP rated 
pilots. 


(10) ATP rated pilots place high worth on time in diversion 
decisions. 

(11) Pilots with good diagnostic scores place less weight 
on approach aids in diversion decisions. 

(12) Pilots with good diagnostic scores place more weight on 
time In diversion decisions. 

(13) The pilots with good diagnostic performance were character- 
ized as knowledgeable about aircraft systems, employed 

few tracks (a track represents a coherent line of ques'^ 
tlons, e.g.| fuel systems), used few Inquiries per track, 
and emphasized time in their destination diversion decisions. 

B. Research Ob.lectlves 

The major objective of the current research was to extend the 
effectiveness of past pilot diagnostic and decision testing through 
the use of computer Interactive terminals employing existing software. 
Such interactive terminal systems provided the researcher with: 

(1) display graphics (e.g., segments of Instrument panels with 
moving Indicators) , (2) real-time capability to allow timing of 
response elements, (3) rapid analysis of pilot inputs and (4) positive 
and/or negative feedback of results as a special experiment. Subject 
Interactions with aircraft controls and displays were programmed 
as well as communication with ATC by selecting information require- 
ments from an array ("menu”) of information sources available. 

The research plan entailed three major research phases embodying 
seven research tasks as shown In Figure 1-1. All of these tasks 
except Task 1 are detailed in the chapters to follow. Task 1, the 
symposium on pilot decision making, has been reported separately in 
Proceedings for the First International Symposium on Aviation 
Psychology (4). A table of contents for the proceedings is reproduced 
in Appendix G of Volume II of this report. 



ORIGINAL PAQK 
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OVERVIEW OF THE RESEARCH 



Figure 1-1 











Chapter II: Development of Computer Aided Formats 


The major task In this research was to design and Implement 
a computer aided testing device for studying pilot diagnosis 
and destination-diversion decision makings The Initial software 
for the computer graphics terminal was designed around the problem 
scenarios, knowledge tes^t, and biographical questionnaire used 
In previous paper and pencil studies (Tasks 2 and 3 of the pro- 
ject proposal). Some new diagnostic scenarios were also designed 
to take advantage of the capabilities of PLATO® (Task 5) . In 
addition, both diagnostic and destination diversion scenarios 
were combined Into a single scenario with added workload (Task 6) . 

A. Program Design 

Appendices A to E In Volume II of this report Include sample 
displays presented to the subject In the course of testing. The 
CIFE data collectlon/subject testing system was built using CDC's 
PLATO® system and the TUTOR programming language. Details on the 
various programming modules are discussed In Appendix F^ 

The goal was to produce a set of programs which could be 
nearly experimenter-free. Considerable effort was expended to 
provide the subject with detailed Instructions via the graphics 
display on exactly what tasks were expected of him. After an 
Initial slgn-on via the terminal keyboard, all subsequent communica- 
tion between subject and computer was accomplished by means of the 
touch sensitive CRT screen. The major elements of the program 
displayed for the subject Included: 
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(1) CIFE Router 

(2) Biographical Data 

(3) Knowledge Test 

(4) Six Diagnostic Scenarios 

(5) The Destination/Diversion Scenario 

(6) The Airport Ranking Exercise 

(7) VOR- Autopilot 

(8) Combined Destination Diversion and Dlagnosi:lc Scenario 
Test (FGCIFE) 

(9) Data Display 
Router 

The CIFE Router Is the main program module which allows access 
to all other modules and to the data files. The router Is accessed 
by a special student sign-on to the PLATO® system. At the completion 
of each separate program module, control Is returned to the Router 
where the subject may select the next module by touching the appropri- 
ate place on the CRT screen. 

Biographical Data 

The first time a subject uses the program he is assigned a 
unique subject identification number and is forced to enter the 
Biographical Data module in order to establish a subject data file. 

If testing is interrupted for any reason, the same subject number 
will permit access to any module without again going through the 
Biographical Data module. All biographical data are displayed in 
the form of multiple choice questions to which the subject responds 
by touch panel. A sample of biographical data questions is con- 
tained in Appendix A. 
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Knowledge Test 


The Knowledge Test module ptesents a series of twenty 
multiple choice questions one at a time. These questions cover 
three subcategories; (1) Engine and Fuel Systems, (2) Electrical 
Systems and Cockpit Instrumentation, and (3) Weather and IFR 
Operations. The Knowledge Test displays are contained in 
Appendix A. 

B. Diagnostic Scenario Operations 

In each of the diagnostic scenarios the subject is told that 
he is flying a Piper Cherokee Arrow and is given a list of equip- 
ment and performance parameters. The next display is then a brief 
paragraph which describes his missioict and the symptoms of a problem 
being encountered. The subject then has a fixed amount of time 
(usually four minutes) in which to seek information and arrive at 
a diagnosis of the problem. Information is available from four 
separate displays which can be called up by the subject at any 
time during his allotted test period. He is not penalized for 
the time required to paint a new display on the screen. 

The information displays Include: . 

(1) instrument panel 

(2) interior Information 

(3) exterior information 

(4) ATC information 

The instrument panel display contains most of the instruments 
and controls found in a Cherokee Arrow. Information is obtained by 
touching the appropriate dial or control. If, for example, a sub- 
ject wants to know his oil pressure reading, he touches the oil 



pressure gauge, shown graphically on the panel layout, and 
its current status is printed for him in the display area. 

It ia also possible to obtain Information concerning control 
movement. For example, if the subject touches the prop rpm, 
a secondary input display permitting him to Increase or decrease 
rpm appears. If he touches either of those boxes, an appropriate 
response message is then printed. The instrument panel display 
is shown in Appendix A. 

The three remaining information displays all work alike. 

A subject touches the box containing desired information and a 
description of that information is printed out for him. Interior 
information refers to inside cabin conditions. This display per- 
mits one to find out about smoke, fluid leaks, unusual sounds, 
etc. which may be in the cockpit. Exterior information includes 
such things as cowling condition, wing .condition, etc. A query 
on wing condition for example might bring forth a response "light 
rime ice visible". ATC information includes general weather and 
navigation aid status information of the type that a Flight 
Service Station or Air Route Traffic ^ntrol Center might provide. 
These data Include such items as forecast winds aloft and freezing 
levels. All three displays, interior, exterior and ATC information, 
are shown in Appendix A. 

Vnien the subject reaches a conclusion Concerning the cause 
of the problem (or he runs out of allotted time) , he pushes the 
"give answer" choice available with each of the Information displays. 
This action routes him to a second choice menu which permits him 
to seek further information in the event that he Inadvertently or 


prematurely puehed "give enewer". 4 eecond puah of "give 
anawer*' than routea him to the lexicon illuatrated in Appendix 
At The subject formulates his diagnosis by touch panel entry 
designating up to nine of the words listed in the lexicon. 

After entering his lexicon response the subject is asked 
a series of questions concerning his judgment of the problem. 

The computer asks for an estimate of how long the plane will 
fly with its existing problem, how critical (scale 1 to 7) the 
problem is, and how confident the subject is in his own diagnosis. 

At that point the proble’n around which the scenario was 
designed is presented to the subject as the correct diagnosis. 

He then is again asked to respond to the time and criticality 
questions in light of his now complete knowledge of the state of 
his aircraft systems. Control tWm passes back to the router 
where a new scenario or other task can be selected. 

A complete time history of all subject data inquiries is 
maintained for each individual subject tested. The particular 
display and the item on '^ 'et display which was queried are noted 
together with the time since initiation of that scenario at 
which the query took place. This time history of information 
search is available through the Data Display module. 

C. Diagnostic Scenario Content 

Diagnostic scenarios one through four are adaptations of 
scenarios previously used in paper and pencil testing. (See 
project report NAS2-10047 (6)). They concern; (1) an oil pressure 
gauge line break, (2) a vacuum pump failure, (3) a broken magneto 
drive gear, and (4) a blocked static port. 


Two new acnaarloe wore ereoted to accompliah certain computer 
aided testing objectives v^hlch were not part of the earlier paper 
and pencil tests. Scenario five is a nearly no-win situation. 

The problem la manifested by a partial power loa& which no amount 
of cockpit experimentation can correct or even totally identify. 

The power loss is assuned to be caused by a broken baffle in the 
muffler which creates a horsepower robbing backpressure in the 
exhaust system* This scenario is designed to explore a subject's 
information Search patterns while eliminating the possibility of 
accidently uncovering the ^ey element of information which uniquely 
identifies each of the four earlier scenarios. Scenario five also 
forms the diagnostic portion of the combined destination diversion 
and diagnostic search task to be described later. 

Scenario six is designed to present symptoms which might be 
Improperly attributed to a mechanical failure when in fact a non- 
mechanical act has caused the problem. This scenario concerns an 
upholstery fire caused by a carelessly discarded cigarette. 

The complete text of each scenario' and the proper diagnosis 
as presented to the subject are reproduced below and in Appendix B. 


Diagnostic Scenario One 


ORIGINAL PAGE 13 
OF POOR QUALITY 


You rnaking a day trip from 
Albany, NY to Burl ington, VT. You 
fly out of Albany at 9:. 00am, claarad 
Victor-91, Burlington. You climb to a 
cruising altiti;da of 7000ft. Aftar 
20 mi nut as of routine IMG flying 
you notice the smell of engine oil. 

How would you diagnose the problem? 


JJur • di agri'oais -»-f the prc-blern was, the f Pillowing 

A small crack developed in the oil line 
feeding the oil pressure gauge. This 
crack reduced the oil pressure reading 
drastically, but did not seriously 
a f f ec t t he act ua 1 1 ubr i ca t i on o f t he 
engine. A small pool of oil began to 
form on the floor of the cabin, pilot's 
side. Assuming that the cracked line 
would not deteriorate quickly into a 
complete break, you were in no immediate 
danger of engine seizure. 


ORiaiMAL PAGE m 
OF POOR QUAUTY 


Diagnostic Scenario 2 


You are making a day trip from 
Augusta, ME to Lebanon , MH ..You, fly 
out of Augusta at 9:00 am, cleared 
Victor 39 to Meets intersection, 
Victor 496 to Lebanon. You climb to 
a cruising altitude of 6000 ft. After 
15 minutes of routine IMG flying in 
i nst rument cond i t i ons , your i nst rument s 
indicate an increase in airspeed and 
steadily decreasing altitude while 
ma i nt a i n i ng 1 eve 1 f I i ght at t i tude . 

How would you diagnose the problem? 


Our diagnosis of the prob 1 em was the fol lowing 


Your vacuum pump failed as indicated by 
the low reading of the suction gauge. 

The vacuum pump drives the attitude and 
d i rect i ona 1 gyros .As the art i f i c i a 1 
horison lest its drive it started to sag 
to the right and you compensated by 
turning left , level ing the artificial 
horizon and putting the plane in a slow, 
descending left bank. The airspeed in- 
crease was due to the slight nose -down 
attitude. 


y 



Diagnostic Scenario 3 


ORIGINAL PAGE IS 
OF POOR QUALITY 


You are making a day trip from 
Ke-sne, NH to Montp«lier,VT. You 
fly out of Keene at 10; 30 am, cleared 
Victor- 151 to Montpelier. You climb 
to a cruising altitude of 5000 ft. 
After 20 minutes of routine cruise in 
IMC your engine suddenly starts 
running extremely rough, shaking the 
whole plane and losing about 20^<» of 
its cruise power. 

How would you diagnose the problem? 


Our diagnosis of the problem was the following 

Your engine suffered a broken drive 
gear in the right magneto. The 
r esu 1 1 ant unt i med ignition cot‘i f I i ct ed 
with t he rema ini ng good i gn i t i on and 
caused the extremely rough engine and 
back f i r i ng . Sw i t ch i ng from 'both' to 
the left magneto would have resulted in 
a smooth running engine with slightly 
less power than normal cruise. 


Dlaanoatic Scenario 4 


ORIGINAL PAGE IS 
OF POOR QUALITY 


’S‘‘ou are making a day trip from San- 
ford, ME to Me»sena,NY, You fly out 
of Sanford at 8:30am, cleared Victor- 
•496 to Lebanon, Victor- 141 to Meeaena 
You climb to a cruise altitude of 6000. 
Rf ter 20 min IMC flying, Boston 
Center instructs you to climb and main- 
tain 10, 000 ft. You acknoL‘iledge and begin 
you?- climb between 1 ayers . R fter 2 min 
of climb, you not i ce your i nd i cat ed a i ?-- 
speed dropping off steadily from I00kts, 

maiTitaiTiing constant pitch attitude. 

# 

How would you diagnose the problem? 


Our d i agnos is of the pr ob 1 em (.uas the f o 1 1 vow i ng 

Rs you climbed through 6500ft, the 
static port froze over as the outside 
air temperature dropped below 32*^ F. 

This caused the airspeed indicato?- to' 
decrease as altitude increased ‘ and the 
VSI and altimeter to read low. 

Severa 1 correct i ve act i ons were 
possible: t-etuT-n to your previous alti- 
tude of 6000ft; open the alter-nate 
static source; break the VSI glass. 



Diagnostic Scenario 5 


ORIGINAL PAGE IS 
OF POOR QUALITY 
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You. are rna.k i T’lg a clay tr i p f r-orn 
RugU5ta,ME to Lebanon , NH. You fly 
out of Rugusta at i0ani, cleared 
Victor 39 to Meets Intersection, 

Victor 496 to Lebanon. You climb to a 
crusing altitude of S 000 ft. Rfter 
20 minutes of routine flying in 
instrument conditions with light to 
moderate turbulence, you notice that 
increased nose-up' trim is required to 
maintain a const ant i nd i ca t ed a 1 1 i t ude 
and that your IRS has decreased 2.0'kts 
from normal cruise. 

Ho>w would you diagnose the problem? 

Our diagnosis of the problem was the following ; 

R baffle was broken in the muffler. 

The broken baffle partially blocked the 
exhaust system causing increased exhau.st 
backpressure. The increa.sed exhaust 
backpres.sure ab.sorbed a portion of the 
ava i 1 ab 1 e liorsepower output from the 
engine. With a constant throttle 
•setting, the p>rop flattened pitch to 
maintain constant RPM causing a 
deer ease i n a i rspeed wh i 1 e a 1 1 i t ude was 
held oo'nstant. Conversely when airspeed 
wa.s held constant altitude decreased due 
to reduced poMjer output ava i lab le^ at the 
pro-pel ler. iTicre-a.sing the manifold 
pres-sure with added thro'ttle permitted 
enough power to be develc-ped to maintain 
altitude at a greatly reduced airspeed. 



Diagnostic Scenario 6 


ORIGINAL PAGa IS 
OF POOR QUALITY 


vpu are makina a day trip from 
ncntpel ier .VT to Banaor.riE with • 
two passenaerE on board. You fly out of 
Montpelier at 1:00pm, cleared radar 
vectors to Wylie intersection, direct 
Auqusta, Victor 3 to Danqor. You 
climb to a cruising altitude of 9000ft. 
After 30 minutes of routine flying in 
instrument conditions with light to 
mcderate turbulence, one pf vpur 
passengers reports smellina a faint 
burning odor. You are unable to detect 
the odor because you have a head cold. 

What is the first thing you would do? 


Our diagnosis of the problem was the following: 


Rear seat carpfc-tina was smolderina. 

The rear seat passenoer lit a 
cigarette shortly after takeoff. When 
he disposed of it in the ashtrc»v, it was 
not completely e.\* t i ngui shed. The 
ciqarettB fell down from the ashtray 
and was beginning to char upholstery 
material. The fire was easily 
extinguished, once recognised and posed 
no immediate danger to the flight. 


D. The Degtlnatlon Diversion Scenario 


The destination diversion scenario was also an adaptation 
of an earlier scenario used In paper and pencil tests. The 
goal was to examine the Information seeking activities and 
decision making strategy of subjects forced to make an enroute 
diversion decision. 

The subject Is presented with a business trip scenario 
involving an IFR flight from Bangor, Maine to Glens Falls, New 
York In a Piper Cherokee Arrow. Area weather charts, a Flight 
Service Station briefing and a flight plan are presented In turn 
as the scenario begins. Once the flight Is underway, a simplified 
low altitude enroute chart Is shown on the graphics display. A 
flashing arrow on that chart depicts the progress of the flight 
as the scenario continues to depict flight progress along the 
Intended path. At a point midway along the planned flight the sub- 
ject Is told that he has experienced an alternator failure and must 
now operate on battery power alone. The maximum expected life 
of the battery Is Insufficient to carry him on to his planned 
destination. The problem now Is to select a suitable alternate 
airport. 

Alternate airports are depicted by number on a further 
simplified enroute chart. The subject may ask for up to six 
Items of Information about any of the sixteen airports depicted 
on the chart: 

(1) bearing and distance 

(2) celling 

(3) visibility 

(4) approach aids 



ATC services 


( 5 ) 

(6) terrain 

He has tm minutes In which to conduct his Infomatloii search 
and select an altematie airport* A summary dl^^play of all 
Information requested is available to the subject anytime he 
wishes to examine It. 

A complete time history of his Information search Is main- 
tained In the compi;iter data file along with his final choice of 
airport. A complete set of graphics displays shown to the sub- 
ject during testing Is shown In Appendix D. A sample of the 
stored data is contained In Appendix C. 

Airport Ranking 

As an alternative way of studying the diversion decision, an 
airport ranking exercise was Created. The ranking 'exercise was 

t 

the alternator failure scenario previously discussed. This module 
accomplishes the same thing by way of computer graphics that was 
previously accomplished by shuffling cards In the paper and pencil 
studies. 

Here the subject is presented with. a matrix of information 
involving sixteen possible diversion airports. Each airport is 
listed together with its ATC services, celling, visibility, time 
to reach and approach aids. The subject ranks these airports from 
most desirable to least desirable. All entries are made by touch 
panel. The matrix is updated with each ranking decision by 
interchanging rows, replacing the current occupant in decision 
r<>w X with the occupant the subject would prefer while moving the 


old occupant In x to the position vacated by the new choice. 

The displays used in this module are shown in Appendix D. 

E. VOR-Autopilot ^ 

The yOR'^autopllot module is a training module designed to 
instruct a subject on the use of the dynamic navigation and 
control capabilities he will later use in the combined destina- 
tion diversion and diagnostic scenarios. The basic control dis- 
play is shown in Appendix E. 

By appropriate touch panel entry a subject can select desired 
heading and altitude commands for his aircraft autopilot. If the 
command heading is not the one currently displayed on the directional 
gyro» the simulated aircraft will begin a standard rate turn toward 
that heading. If the altitude selected is not that currently 
stored in the instrument panel display, the simulated aircraft 
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will begin a pre-programmed rate of climb (or descent) . At the 
same time that Instrument indications are being updated, the simu- 
lated aircraft continues to move through space. That movement 
is reflected by the tunable VOR heads contained on the display. 

The subject can determine his current position in space by 
selecting appropriate VOR frequencies and centering the needles. 

The adraissable VOR's are depicted on an accompanying simplified 
low altitude chart. 

Each subject is given a few brief training exercises involving 
turns to heading and locating the aircraft relative to VOR stations. 
This training exercise does not refer to any particular problem 
scenario but rather is intended to familiarize him with the new 
dynamic system. 


F. C^rablnlng Deatlnatlon Dlveralon With Diagnosis 

The combined scenario described In Figure 2-1 was designed 
to meet two purposes; 

(1) It was a "no win" problem, l.e. no pilot would really 
be able to find the true cause from the symptoms pre- 
sented* Hence, It would avoid a pilot stumbling Into the 
key information Item and force the pilot to examine all 
possible hypotheses. The diagnosis portion was the 

same problem previously discussed as scenario (1*5. 

(2) It combined on the touch CRT both the destination diversion 
decision and the problem of diagnosis. 

Appendix E shows the displays used In this scenario. This 
Includes a simulated low altitude chart. A program has been 
developed for FLATOS^ which locates the aircraft relative to the VOR 
site, allows heading changes and allows the position of the air- 
craft to be determined from VOR radlals as the aircraft moves at 
some selected speed and heading. It should be noted that attempts 
to develop positional awareness are used throughout. These Include 
position reports to ATC based on dual VOR's location assessment. 


auto pilot heading change requirements, new clearances and even 
a concerned passenger requesting a return to the departure airport. 
This scenario employs some dynamics, i.e. loss of altitude with 
neutral forces on the yoke and attendant changes in VSI and 
airspeed with control inputs. As the flight progresses the VOR 
needle shows a deflection. The scenario permits communication with 
ATC and allows for declaring emergencies. 


Consult sttached simplified altitude chart. 

You are on an IFR flight from Utah Municipal Airport to 
Haven Count Airport. You depart on V-UO at 6000 ft. In your 
Cherokee Arrow (H123B) which is equipped with a 3-axls auto- 
pilot. There Is a NOTAM out which reports that Colorado VOR 
Is out of service during the period you plan to navigate. 

Navigate using Ohigh and California VORs . You have been enroute 
60 minutes from Utah Municipal Airport. You arc on die gauges 
but the ride Is smooth. Weather briefing Indicated that winds 
at 6000 were expected to be light and variable. 

You have one passenger aboard. 

Weather at: 

Haven County Airport ■ 2000 & 5 
Ohigh - 1000 & 3 

Wind Falls • 1000 & 3 by a C-172 
(10 Minutes Ago) 

Cleve Center calls and reports radar contact Is lost. Please 
report present condition. 

Clearance 

ATC Response: 

N123B, thanks for the position report. 

Here Is your new clearance: 

proceed direct California VOR direct 
Haven County Airport at 6000. 

There will be opposite traffic at 5000 . . . maintain 6000. 
Please confirm your new heading and altitude after your turn. 
Scenario Change 

While practicing hand flying with your autopilot disengaged, you notice 
that Increased nose-up trim is required to maintain a constant 
indicated altitude and that your IAS has decreased 20 kts. from normal 
cruise. 

Your passenger notes this problem, and suggests that you turn back to 
Utah Municipal. 

Determine the nature of the problem, and your destination decision. 


Figure 2-1: Combined Diagnosis and Destination Diversion Scenario 


The purpose in combining diagnosis and destination diversion 
problems into a single scenario was outlined in Task 6 of the 
original proposal. The combined scenario was created to pro” 
vide a more realistic framework for tests which would begin to 
approach the fidelity of the earlier GAT simulations, in the sense 
of dynamic position information and added workload. The research ques- 
tion was whether PLATO^ lends itself well to simultaneous navigation 
and diagnosis even though no control manipulation skills beyond 

selection of heading and altitude commands were possible. 


Chapter HI; Raaulta of Computer Aided Tearing 


Two typca of analyaes were made baaed on the data generated 
through computer aided testing. The first type of anaXyala 
waa baaed on the power of the computer to generate a variety of 
summary statlstlca. Here such standard techniques aa regression 
analysis t t- tests, chi-square testa and frequency counts were 
used on a variety of different combinations of subject data. 

The intent was to Isolate Important performance measures and 
classify groups of subjects. 

The second type of analysis required the generation of crea- 
tive graphical aids which permit a particular subject's informa- 
tion seeking strategy to be absorbed at a glance. By comparing 
performance across subjects it became possible to identify several 
distinct search strategies which were not apparent from the more 
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formal statistical teats. The graphical aids used to depict informa- 
tion seeking behavior during diagnosis testing were the pilot Informa- 
tion plot (PIP) and schema diagrams. The aid used to depict the 
destination diversion information search was the destination 
information graph (DIG) . 

A. Depiction of Diagnostic Information Seeking Patterns For 

Individual Subjects 

Figures 3-2 through 3-5 depict a way to view subject informa- 
tion seeking patterns during diagnosis testing. Sources within logic 
tracks are identified for each scenario. The pilot information 
plots (pip's) are a quick way to visualize: 



(a) th« numbar o£ tracks A employed 

(b) the order of inquiries within and between tracks 

(c) the time between inquiries 

(d) the number of track returns and the information resampled 

Using these PIP's various Information seeking strategies can 

be observed. For the suction failure problem as shown In Figure 3’‘1, 
Figure 3-2 depicts a subject with a logical and efficient approach 
to diagnosis. Figure 3-3 depicts an almost random Inquiry which 
leadf^ to no logical conclusion. For most scenarios there is a 
key element or piece of information to identify the problem, e.g. 
low suction and vacuum failure. Figure 3-4 shows a subject with 
the key piece of Information but who still does not recognize the 
correct answer. Figure 3-5 clearly indicates a subject using a 
systematic approach but an Incorrect one. In this case the, sub- 
ject believes ice to be the causal factor for the symptoms presented. 

Based on PIP analyses, idealized information searching can 
be hypothesized. The ideal pilot first confirms the symptoms 
given him. He then establishes whether his engine status is 
threatened by whatever cause lies behind the symptoms. Usually 
oil temperoture and pressure and manifold pressure suffice to rest 
this condition. Next he makes two or more hypotheses as to the 
cause, and makes a determination of the plausibility of these 
hypotheses with a minimal number of inquiries within the appropriate 
tracks. He rarely needs to go over old logic tracks sampled, 

Finally, given a logical cause of the symptoms (usually from the 
key Information element) , he will often make sure alternative 
hypotheses are still not viable by additional Information inquiries. 

*a track is a coherent line of questioning focused on an aircraft 
subsystem, e.g., internal engine condition. 


SCENARIO 


You are making a day trip from Auguata, ME to 
Lebanon^ NH. You fly out of Augusta at 9:00 a.m. , 
cleared Victor 39 to Neeta Intersect^loti, Victor 
496 to Lebanon. You climb to a cruising altitude 
of 6000 ft. After 15 minutes pf routine IMG fly- 
ing in Instrument conditions, your Instruments 
indicate an Increase in airspeed and steadily 
decreasing altitude while maintaining level flight 
attitude. How would you identify your problem? 


Our Diagnosis of the Problem was the Following ; 

Your vacuum pump failed as indicated by the low 
reading of the suction gauge. The vacuum pump 
drives the attitude and directional gyros. As the 
artificial horizon lost its drive it started to sag 
to the right and you compensated by turning left-, 
leveling the artificial horizon and putting the plane 
in a slow, descending left bank. The airspeed increase 
was due to the slight nose-down attitude. 


Figure 3-1; Suction Failure Problem and Diagnosis 
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RESPONSE: Gyro Broken 

CONFIDENCE IN RESPONSE: 7 

CRITICALITY: 5 

TOTAL FLIGHT TIME: 100-300 Hrs. 

TOTAL DIAGNOSIS TIME: 78 Sec. 

RATING: Private - VFR 

KEY TERM: 

TOTAL NO. INQUIRIES: 9 

KNOWLEDGE SCORE: 60% 

*Time Between Inquiries 

Figure 3-2. Pilot Information Plot For Scenario il2 Subject §53 
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RESPONSE: Elevator Ccmtrol Pressure ' 

CONFIDENCE IN RESPONSE: 7 

CRITICALITY: 1 

TOTAL FLIGHT TIME: 100 Hrs. 

TOTAL DIAGNOSIS TIME: 2U Sec. 

RATING: Private - VFR 

KEY TERM: 0Y 
TOTAL NO. INQUIRIES: 20 

KNOWLEDGE SCORE: 50% 

*Time Between Inquiries 

Figure 3-3. PILOT INFORMATION PILOT FOR SCENARIO it! SUBJECT //67 
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RESPONSE; Pitot Ice 
CONFIDENCE IN RESPONSE: 6 

CRITICALITY : 3 

TOTAL FLIGHT TIME; 501-1000 Hrs. 

TOTAL DIAGNOSIS TIME: 127 Sec. 

RATING: Commercial^IFR 

KEY TERM: N @ 

TOTAL NO. INQUIRIES: 19 

KNOWLEDGE SCORE: 50 % 

*Time Between Inquiries 

Figure 3-4. Pilot Infonnation Plot For Scenario {12 Subject {155 
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RESPONSE: Pitot Static Prop Ice 

CONFIDENCE IN RESPONSE: 6 


CRITICALITY: 6 


TOTAL FLIGHT TIME: 501-1000 Hrs. 


TOTAL DIAGNOSIS TIME: 123 Sec. 


RATING: Conmiercial -IFR 


KEY TERM 


! 0 


TOTAL NO. INQUIRIES: 16 


KNOWLEDGE SCORE: A OX 


*Tinie Between Inquiries 


Figure 3-|. Pilot Inforraation Plot For Scenario //2 Subject r5A 
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B. Smmnarlzlng Diagnostic Information Search Patterns Graphically 

Schema theory provides a method for investigating the under- 
lying logical structure (schema) which provides a mental outline 
by which a pilot may organize information! A pilot has some know- 
ledge of the events that typically occur in a critical in-flight 
event and the order in which they take place. The completeness of 
his knowledge will determine the ease with which he can comprehend 
and summarize diagnostic information. By separating "correct" 
diagnoses from "Incorrect" diagnoses one may be able to develop 
guidelines from schemata which highlight efficient information 
search strategies. 

The schema diagrams developed for this research plot each 
item of information sought by the frequency with which it is 
requested and the median order percentile in which that item 
appears in the total stream of information requests. The 
technique is patterned after that discussed by Gelselman and Samet 
(3). 

For example, on scenario 1 subject 64 asked for 26 separate 
items of information over a period of 205 seconds. Cylinder head 
temperature was the third item of Information sought and that was 
requested at 8 seconds into his search. In terms of order, this 
information was in the 11.5 percentile order position and the 3.9 
percentile time position. By performing similar calculations for 
each subject it was determined that 90 percent of the subjects in 
scenario 1 with the correct. diagnosis asked for cylinder head temper- 
ature (CHT), that the median order percentile for their requests 
was 34.5. These data were then plotted on two schemata, one 
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showing frequency versus order and the second showing frequency 
versus time. 

By grouping items of Information Into common subject matter 
clusters^ It Is possible to develop a two-dimensional hierarchical 
outline of subordination and sequential order of information. 

Level of subordination Is measured by the percent of pilots who 
Include that Information In their search. Output position Is 
standardized by computing a median output position percentile 
for each Item of Information which was sought by at least one 
subject. . 

Figure 3-6 shows a schema diagram for successful subjects on 
the first scenario. This scenario involved a cracked oil pressure 
gauge line behind the instrument panel. Positive confirmation of 
the problem was possible by noting the reduced oil pressure and 
the presence of fluid leaks in the cockpit. Roughly 90 percent of 
the subjects Included oil pressure, oil temperature, cylinder 
head temperature and fluid leaks in their search, although fluid 
leaks did show up late In the order of information (cluster I on 
the schema diagram). Another sizeable portion, 60 percent, appear 
to be searching for evidence of an oil system failure in front of 
the firewall (cluster 2) . A much smaller minority of approximately 
10 percent appear to be concerned about general engine health 
(cluster 3) . The Information in cluster 4 which appears late in 
the search process for some of the subjects seems to be directed 
toward localizing the source of the smell of hot engine oil, e.g. 
is it in or outside of the cabin. Note that even though all of the 
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Subjects; 42, 43, 46, 47, 51, 52, 53, 54, 55, 56, 57, 59, 60, 61, 64, 67, 68* 80 


Freq. (%) 
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Figure 3-6: 



Schema For Subjects In Scenario H With Correct Diagnosis 

(See Figure 3-21 for the list of symbols used in schemata on Page 89) 
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subjects In this schema solved the problem, the median percentile 
position for the key Item "fluid leaks" was only 78. Evidently many 
sought further confirmation even after having the key. As identified 
earlier this Is an ejcample of pilots checking on alternative hypotheses. 

C. Grading Diagnostic Results 

Correctness scores, which were used to measure a subject's 
diagnostic performance, were related to a simple five point scale. 

Each of the four basic scenarios could have been unambiguously 
diagnosed if the subject asked for a certain key piece of Informa- 
tion. For example, in the oil pressure gauge line leak scenario 
the key element was "fluid leaks". In the vacuum pump failure 
scenario the key element was "suction". In the magneto gear fail- 
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ure scenario the key element was "left mag". In the static port I 

icing scenario the key element was "alternate static source". | 

Correct answers were awarded a grade of five, four, or one, depend- | 

Ing upon whether the key element was asked for, whether the subject 
was on the right track or whether there was no apparent connection 
between the information sought and the correct response. 

Incorrect answers which could at least be partially supported 
from a logical search pattern were given two points. Incorrect and 
illogical searches were graded zero. This scheme is summarized in 
the table below. 

Table 3-1. Diagnostic Scoring 
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Conclusion 
Inf ormatlorr---»i». 

Correct 

Answer 

Incorrect 

Answer 

Incorrect 
and Illogical 

Key 

5 

2 

0 

No Key But 




Right Track 

4 

2 

0 

Never on 




Track 

1 

2 

0 
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The relative grade frequenciea observed by scenario were 
a» follows: 


Table 3*2 


Observed Relative Frequency By Scenario 


Information 

Conclusion 

Correct 
Answer 
% of all 
subjects 

Incorrect 
But Logical 

Incorrect 
and Illogical 

Key 

Scenario Ifl 

52.4% 

0% 

9.5% 


Scenario i\2 

23.8% 

2,4% 

4.8% 


Scenario f/3 

63.6% 

0% 

3.0% 


Scenario //4 

43.2% 

2.7% 

0% 

No Key But 

Scenario 

4.8% 

19.0% 

14.3% 

Right Track 

Scenario 1^2 

2.4% 

2.4% 

4.8% 


Scenario //3 

3.0% 

9.1% 

6.1% 


Scenario #4 

2.7% 

29.7% 

13.5% 

Not on 

Scenario //I 

0% 

■ 0% 

0% 

Track 

Scenario //2 

0% 

4.8% 

54.8% 


Scenario //3 

0% 

0% 

15.2% 


Scenario //4 

0% 

8.1% 

0% 


From these data it would appear that scenario 3 was the 
least ambiguous of all. Nearly 65 percent of the subjects taking 
the mag failure scenario found the key and correctly Identified 
the problem. Scenario 2, the vacuum pump failure appears to be 
the most difficult. Nearly 55 percent of the subjects taking 
that scenario drew conclusions which could not be supported by the 
data they collected. In most cases they seemed to have an ice 
fixation and blamed their symptoms on a frozen pitot-static system 
in spite of no collaborating evidence. When faced with a genuine 
frozen static port scenario they were much more logical with only 
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13 percent drawing conclusiona which could not be aupported by data. 
This may reflect the fact that many of these subjects seemed 
obsessed with Ice problems and blamed Ice for any conflicting 
flight Instrument readings. 

D. Depiction of Destination Diversion Information Seeking Patterns 

Appendix D presents the CRT displays given to the subject 
for the destination diversion phase of testing. In this scenario 
the pilot loses his alternator In IMC conditions beyond the range 
of his destination or departure airports. He must choose between 
alternate airports with different attributes. Such attributes 
Include celling, visibility, bearing and distance, navigation aids, 
presence of ATC support and terrain. In this test there Is no 
absolute answer; rather, the test seeks to see how pilots weigh 
Information about alternative airports. 

*, 

The Destination Information Graph (DIG) provides a quick way 
to summarize the Information seeking style of an Individual sub» 
ject. The DIG's portray; 

(a) the type of Information sought 

(b) the frequencies with which Information Items are queried 

(c) the order in which airports are considered 

(d) the ultimate choice of which alternate airport Is selected 
and how much data decision required compared to air- 
ports considered but not selected 

Figure 3-7 Illustrates a typical DIG. This particular subject 
asked for Information on only three airports, although his search 
was nearly total over the Information available for each. The air- 
port selected, number 6, was the first one he searched. 
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Subject #67 
Time Lapsed: 134 

Sec./Port Selection: 13.33 

Sec. /Inquiry; 5,875 
Airport Selected: #6 





Figure 3-7. Destination-Diversion Quick Decision 
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E, Subject Background 

A total of forty-two subject! participated In the conputer 
aided testing (CAT) program. All were rated pilots ranging from 
Private to ATP. Four subjects were used for special purpose 
testing leaving thirty-eight who participated in full sets of 
CAT experiments. Forty-two percent were private pilots, forty- 
two percent were commercial pilots and sixteen percent held the 
ATP rating. Eighty-eight percent of the private pilots were not 
instrument rated. This group was recruited to investigate whether 
or not the CAT scenarios could separate performance of pilots with 
IFR versus VFR experience. 


Total flying time among subjects ranged from fifty hours 
to 15,000 hours, the average being 1,654 hours. Their single-engine 
experience ranged from fifty to 7,500 with an average of 933 hours. 
Instrument flying experience, which In the ca^e of non-rated pri- 
vate pilots included elementary hood time, averaged 257 hours. 

The most experienced instrument pilot reported 1,500 hours of instru- 
ment flying experience. Histograms reflecting this broad range of 
flying experience among subjects are shown in Figures 3-8,. 3-9, and 


3-10. 

Knowledge survey scores ranged from twenty-five to eighty percent 
with an average of 56.6. Although individual subjects were weaker 
in some areas than others, the average percent correct by category 
across all subjects was nearly constant. The distribution of know- 
ledge scores is given by Figure 3-11. 

A brief summary of pertinent biographical Information for 
each subject is given in Table 3-3. A complete subject data record, 
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Figure 3-8r Distribution of Total Flying Experience 
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Figure 3—9: Distribution of Single Engine Flying Experience 
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Figure 3-10: Blstrlbucion of Ins tr intent; Flying Experience 
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Figure 3-11; Distribution of Knowledge Scores 



including response to all scenarios is available In the Master 
Data Table I Table 3-5. A description of the variables contained 
in the Master Data Table is available in Table 3-4. 

F. Diagnostic Performance 

Means and standard deviations for all performance variables 
are listed in Table 3-6. Correctness scores are portrayed in 
Figure 3-12. This histogram illustrates an extremely wide spread 
in performance. Nearly eight percent of the subjects completely 
missed every scenario for a net score of zero while only three 
percent were able to achieve a perfect correctness score. The 
average percent total correct was 47. Individual scenario scores 
are more enlightening. Out of a maximum of five points for each 
scenario, the averages are 3>31, 1.46, 3.33, and 3.00 for scenarios 
one through four respectively. It is obvious that scenario two, 
the vacuum pump failure, posed significantly more problems In 
diagnosis than did the others. This is especially noteworthy in 
view of the current failure rates in general aviation vacuum pumps. 
These data seem to Indicate that pilots may have difficulty In 
recognizing the symptoms of such failures. 

Regression Ataalysls 

In order to investigate what factors predict performance 
measures, a series of stepwise regressions were performed. These 
are listed in Table 3-7 and 3-8. Each table Indicates the dependent 
variable, R , N, and three sets of independent variables (biograph- 
ical and experience, knowledge, independent performance variables) 
from which the predictor variables ate chosen. Table 3-7 indicates 


the regression models attained for the correctness scores on 
scenarios one through £our< Predictive variables Included 
knowledge survey score, number of Inquiries, number of tracks, 
and the mean delta (decision and response) time. Maximum 
R-square improvement was chosen. A significance level (a) of 
0.1 was chosen. Four subjects' (69-73) time data had to be 
deleted since they were given unlimited time to complete a 
protocol analysis. The MAXR stepwise technique employed by 
SAS will eliminate a subject's data from consideration If that 
subject has any missing values for the variables included in 
the regression procedure. This accounts for the removal of 
seventeen subjects from the first set of regression models shown 
in Table 3-7. 

The general strategy was to include all predictor (independent) 
variables, even those which could be related to each other, e.g. 

DIFTT (total tracks - total unique tracks) and total tracks and 

i 

total unique tracks. However, no variable could be Introduced 
if it had been derived from another significant predictor variable. 

On the other hand, no predictor variable which formed part of the 
dependent variable was allowed in the regression. For example, since 
TOTCORR-Cl + C2 + C3 + C4, a predictor variable for TOTCORR could not be 
introduced which contained either Cl, C2, C3, or C4, e.g., Z^, 

(Z^ ■ total correct/total tracks). The reader is referred to 
Tabj.e 3-4 for complete definitions of symbols and terms. 

The strategy employed in deriving the regression models pre- 
sented in Table 3-8 is slightly different from that described 
above. The objective here was to condense the candidate predictor 
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Figure 3-12: Distribution of Diagnostic Scores 
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Table 3-5. Master Data Table (continued!) 
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Table 3-5. Master Data Table (continued) 
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Hat In order to allow all forty-three aubjects' data to be 
conaldered in the ref resalon analyala. Since the majority 
of subjects completed scenarios 2 , 3, and 4, all calculations 
for the independent performance measures were limited to these 
three. Given the criteria just mentioned above, only knowledge 
score and total correctness (TOTCORR >€2+03+ C4) can be 
appropriately modeled. 

When correctness scores on individual scenarios are examined, 
C1-C4, several interesting conclusions appear. Knowledge subscores 
are good predictors for 02 and C4 only. A single experience vari- 
able, source of Instrument training, shows up in 03 prediction. 

The total number of unique tracks and the total number of times 
the subject was on the correct track are positive predictors in 
either scenarios 1 or 2. The lower DIFTT (total tracks - total 
unique tracks), the higher the correctness score is on scenario 1. 

The correctness score on scenario 3 increases as the total number 
of inquiries decreases. As might be expected, the lower the 
variance on delta (decision and response) time, the higher the 
correctness score on scenario number 4. What was surprising 
was the absence of such occurrences over all four correctness scores. 
This may be due in part because good performance on one scenario 
does not necessarily translate into good performance on all four 
scenarios as verified by the learning experiment. 

For Zj. the ratio of total correct to total number of tracks, 
it was revealed that as both DIFTT (TOTRAK - UNIQTRAK) and the 
total number of inquiries diminish, rises. A pilot’s source 
of Instrument training seems to augment Z^. Both and the total 
number of inquiries are negative predictors of mean time between 
inquiries (MDELTAT) . 
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PREDICTOR BIOGRAPHICAL INDEPENDENT 

VARIABLE & EXPERIENCE KNOWLEDGE PERFORMANCE VARS. 




1 

N 


1 


j 



SET TT 
SO AGE 

CATSCRl SCORE 

CATSCR2 

CATSCR3 

Ci-C4, INPTRT, DIPT!, 

MEANl-4, VARi-4, TOTINQ, 
TOTRAK, UNIQTRAK, CORINQ, TOCT 
MDELTAT. TOTCOR. ET. VDELTAT 

Cl 

,65 

26 



C2 (.0031)* 
UNIQTRAK (.0001) 
DIFTT (.0847)* 
C3 (.0072)* 

C4 f.0600)* 

C2 

.80 

26 


CATSCR2 (.0527)* 
CATSCR3 (.0011) 

TOCT (.0014) 

C4 (.0024)* 
VARl (.0011) 
MEAN3 (.0001)* 

C3 

.93 

26 

1 

SO (.0001)* 


Cl (.0536)* 
UNIQTRAK (.0001) 
i TOTINQ (.0021)* 

MEANl (.0169) 
VAR4 (.0162)* 

C4 

.66 

26 

i 

1 

1 

i 

CATSCR3 (.0044) 

i 

G2 (.0066)* 
MEAN 1 (.0139) 
MEAN 2 (.0103)* 
VAR 4 (.0366)* 

ZT 

.79 

26 

SO (.0144) 

1 

CATSCRl (.0550)* 

DIFTT (.0696)* 
TOTINQ (.0259)* 
MEAN2 (.0195) 
VAR2 (.0739) 

DELTA! 

.64 

26 

SO (.0122) 
AGE (.0357) 


C3 (.0041) 

ZT (.0016)* 
TOTINO (.0091)* 


*NEGATIVE B VALUE 
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Referring to Table 3-8 and recalling the procedure utilized 
in performing this set of regression analyses, certain Intuitive 
results appear. The total number of unique tracks, the total 
number of times the subject Is on the correct track, and the 
number of single engine hours are all positive predictors of 
pilot's knowledge survey score. The total number of tracks and 
knowledge score are Inversely related. 

As the total number of tracks and the total number of times 
the subject Is on the correct track Increase, total correctness 
(TOTCORR) will also increase. Total correctness will rise as 
DIFTT (total tracks - total unique tracks) decreases. 

Tests On Data Partitions 

Table 3-9 labeled "Summary of T-Test" shows a series of tests 
on extreme partitions of major Independent and key dependent measures 
to determine If dlfferenc^a might exist between them. These 
extreme "cuts" were obtained from a cumulative frequency table 
furnished by SAS. The splits or cuts were made at the lower 23 
percent quartHe and at the upper 75 percent quartile on biographical 
measures involving flying time and age,, plus two performance measures, 
SCORE and TOTCOR. For example, the cumulative frequency table 
reveals that for this group of subjects the lower 25 percent have,, 
at most, only one hundred hours of single engine flying experience, 
while the upper seventy-five percent have greater than 1,001 hours 
of single engine experience. Partitions were also made according 
to pilot classifications, i.e. private versus non-private, VIR rated 
versus IFR rated, military versus civilian, high recency versus 
lew recency, and pleasure versus non-pleasure. 


original 
OF POOR 


Table 3-8. SUMMARY OF STEPWISE PREDICTIONS 


-based on Seen 2,3,4 


PREDICTOR BIOGRAPHICAL INDEPENDENT 

VARIABLE & EXPERIENCE . KNOWLEDGE PERFORMANCE VARS. 


1 

D 

N 




SCORE 

fSl 

35 

SETLOG (.0034) 


TOTRAK (.0009)* 
UNIQTRAK (.0109) 
C3 (.0776)* 

TOCT (.0078) 

TOTCORR 

f 

,48 

35 


CATSCRl (.0191)* 

TOTRAK (.0934) 
DIFTT (.0125)* 
TOCT (.0004) 


*NEGATIVE B VALUE 
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Table 3-9 Summary of T-test 


Teat Variables 


Data partitions on 




Higher 

N 

Biographical & Experience 




Valued 

for spec. 

and Key Dependent Measures 


Variable 

Prob>|Tl 

Group 

variable 




CATSCRl, 

TOTRAK, 

INPTRT, 

VARl-4. 

, CATSCR2, CATSCR3, SCORE, C1-C4 , TOTCOR 
UNIQTRAK, DIFTT, TOGT, CORINQ, TOTINQ, 
SETLOG, TTLOG, MEANl-4, MPELTAT, 

SO. ZT 

Low Single Engine Hrs. vs. 





6 

High Single Engine Hrs. 


CATSCR2 

(.0361) 

(1) 

10 

(0) 100 

(0) 10 

CATSCR3 

(.0478) 

(1) 

10 

A 

(1) > 1001 

(1) 11 

SCORE 

(.0148) 

(1) 

10 

A 



SETLOG 

(.0001) 

(1) 

11 

f. 



TTLOG 

(.0001) 

(1) 

u 

11 






10 



MEAN2 

(.0365) 

(1) 

9 






10 



VAR2 

(.0391) 

(1) 

9 

Low Total Hrs. vs. 



‘ 


15 

High Total Hrs. 


CATSCR2 

(.0176) 

(1) 

8 






15 

(0) < 300 

(0) 19 

CATSCR3 

(.0958) 

(1) 

8 






15 

(1) > 2001 

(1) 9 

SCORE 

(.0624) 

(1) 

8 






11 



Cl 

(.0367) 

(0) 

6 




* 


15 



SETLOG 

(.0001) 

(1) 

9 






15 



TTLOG 

(.0001) 

(1) 

9 

Private vs. Non-Private 





16 

Rating 


CATSCR2 

( . 0090) 

(1) 

21 

16 

(0) Private 

(0) 16 

CATSCR3 

(.0995) 

(1) 

21 






16 

CD Commercial & Air 

(1) 22 

SCORE 

(.0334) 

(1) 

21 

Transport 





13 



TOTRAK 

(.0761) 

(0) 

16 






13 



UNIQTRAK (.0745) 

(0) 

16 






16 



SETLOG 

(.0001) 

(1) 

22 






16 



TTLOG 

(.0001) 

(1) 

22 
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Table 3-9. (continued) 
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Civilian vs. Military 






31 

(0) Civilian 

(0) 

32 

C2 

(.0674) 

(0) 

6 

(1) Military 






24 

(1) 

6 

TOCT 

(.0149) 

(0) 

5 







32 




TTLOG 

(.0021) 

(1) 

6 



t 




24 




VARl 

(.0427) 

(0) 

4 







27 




VAR3 

(.0100) 

(0) 

5 

Low Total Correct vs. 
High Total Correct 






7 

(0) < 6 

(0) 

11 

CATSCR2 

(.0768) 

(1) 

9 







3 

(1) > 12 

(1) 

9 

Cl 

(.0279) 

(1) 

9 







10 




C2 

(.0453) 

(1) 

9 







6 




C3 

(.0013) 

(1) 

9 







6 




C4 

(.0134) 

(1) 

9 







4 




MEANl 

(.0150) 

(1) 

8 







10 




MDELTAT 

(.0629) 

(1) 

8 







3 




ZT 

(.0219) 

(1) 

9 

Low Percentage Correct vs. 
High Percentage Correct 






10 

(0) < .35 

(0) 

11 

C2 

. (.0282) 

(1) 

10 

7 

(1) > .60 

(1) 

10 

C4 

(.0145) 

(1) 

/ 

10 

• 






6 




CORINQ 

(.0660) 

(1) 

8 







6 




MEANl 

(.0058) 

(1) 

7 







6 




VARl 

(.0987) 

(1) 

7 







5 




ZT 

(.0153) 

(1) 

8 


Table 3-9. (continued) 


High Recency vs. 

Low Recency 

(0) Last 30 days (0) 23 

(1) > 2 years (1) 10 


Pleasure vs. Non-pleasure 


(0) Airline 


Military 

(0) 17 

Hired 

(1) 21 

Business 


(1) Pleasure 
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CATSCR2 

(.0403) 

(0) 

CATSCR3 

(.0092) 

(0) 

SCORE 

(.0017) 

(0) 

C3 

(.0252) 

(0) 

TOTRAK 

(.0147) 

(1) 

DIFTT 

(.0100) 

(1) 

TOTINQ 

(.0017) 

(1) 

SETLOG 

(.0171) 

(0) 

TTLOG 

(.0277) 

(0) 

MEAN2 

(.0580) 

(0) 

ZT 

(.0661) 

(0) 


18 

10 

18 

10 

18 

10 

18 

9 

14 

8 

14 

8 

23 

10 

19 

10 

19 

10 

19 

10 

14 

7 


17 

20 

17 

20 

15 

14 

17 

21 

17 

21 




Thirty-one performance and experience variables were chosen 
as candidates for the t-test. The objective was to see If there 
was any significant difference between the means of the twenty- 
five percent quartlle and the seventy-five percent quartlle for the 
performance or experience measure In question. 

Separating single engine experience revealed a higher know- 
ledge survey score for pilots with over 1,001 hours versus those 
with less than one hundred hours. Total flying experience (< 300 
hours versus > 2,001 hours) appeared to exhibit the same relation- 
ship to performance as did single engine flying experience. Splits 
on ratings revealed that commercial and ATP pilots tend to possess 
greater knowledge than do private pilots. Private pilots tend to 
use a larger number of total tracks and total unique tracks than 
do commercial or ATP pilots. 

For training partitions, military training led to smaller 
variances on delta times (decision and response time) for scenarios 
1 and 3. Civilian pilots tend to have fewer hours than do pilots 
with military training, However, civilian pilots attained a higher 
correctness score on scenario 2 (vacuum pump failure) . The total 
number of times the subject is on the correct track is also higher 
for the civilian pilots. 

A split on IFR versus VFR rated pilots disclosed that IFR airmen 
retain a higher knowledge survey score and greater total correctness 
than do VFR airmen. 

When performance measures are split to get profiles of high 
score pilots versus low score pilots, some independent performance 
effects are noted. Higher knowledge scores (>65%) are associated 


with: 



(a) higher single engine hours experience 

(b) higher total flying experience 

(c) higher correctness scores 

(d) higher mean delta times 

When the top and bottom quartile on correctness scores are 
examined, added independent performance results begin to appear. 

High total correctness (>12; >60/^) is characterized by the following: 

(a) higher knowledge scores on cockpit instrumentation 
and electrical systems 

(b) a greater number of inquiries on the correct track 

(c) higher mean delta times 

tho « pilots who submitted an instrument flight rules plan in 
the last thirty days are characterized by: 

(a) higher knowledge scores 

(b) a higher correctness score on scenario 3 

(c) a higher ratio of total correct to total number of tracks (Z^) 

(d) greater total flying experience 
(«) fewer total tracks 

(f) fewer DIFTT 

(g) fewer total inquiries 

as opposed to those who have not done so in the past two years. 

Pilots who fly mainly for pleasure tended to score better on 
scenario 1. However, this class of pilots showed less knowledge 
and less experience than people who fly mainly for airline, commercial, 
business, or military purposes. Airmen over fifty years of age 
achieved better knowledge scores than those under twenty years of age. 


PIP Analysis 


A Pilot Infonnatlon Plot (PIP) was created for every 
subject on each diagnostic scenario attempted. These charts 
serve much the same purpose as the flow process charts long 
in vogue by industrial engineers. They provide a quick visual- 
ization of the Information gathering style of a subject. One 
can tell at a glance which subjects have a systematic approach 
to collecting Information and which seem to "shotgun" across 
the panel In search of any clue to the problem at hand. For 
example, one can see extremes In subject performance on scenario 2 
from the PIPs presented and discussed In chapter 2 of this report. 

It Is also possible to compare performance of a single sub- 
ject across all scenarios. For example, subject 42 Is depicted 
in Figure 3-13 through 3-16. This subject has a very systematic 
search pattern which Is similar for all four scenarios. He sought 
multiple data on a single track before switching. In three of the 
four he found the key element of Information but In each case he 
elected to seek additional information to verify what at that 
point must have been an almost certain conclusion. 

Subject 57 on the other hand exhibits an almost random data 
acquisition pattern. Figure 3-17 through 3-20 show evidence of many 
changes in tracks and a great number of inquiries. It appears as 
if this subject has no hypotheses to test but rather collects all 
the data possible in the time allotted before drawing any conclu- 
sions. This subject also returns to tracks and even to items within 
track many times. Yet in spite of this total immersion in data 
the subject did correctly diagnose two of the four scenarios taken 
simply by accidently discovering the key information element. 


Figure 3-13: Pilot Information Plot For Scenario //!» Subject HZ 
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Figure 3-14: Pilot In£omatlon Plot For Scenario //2, Subject H2 
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Confidence in Response: 

Criticality : 

Total Flight Time: 100-300 Hrs. 

Total Diagnosis Time: 241 Sec. 

Rating: Private 

Key Term: @ Y 

Total No. Inquiries: 13 

Knowledge Score: 55% 







































ORIGINAL PAQI » 
OP POOR QUALITY 




Xcio 



Vacut’Ai 




AW 



Response : 

Confidence in Response; 

Criticality; - — 

Total Flight Time; 100-300 Hours 
Total Diagnosis Time; 185 Sec. 
Rating; Priva^ 

Key Term; N © 

Total No. Inquiries; 13 
Knowledge Score ; 55% 


Figure 3-16; Pilot Information Plot For Scenario Subject //42 
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Figure 3-17; Pilot Informatlor. Hot For Scenario Subject //57 






-84 


Figure 3-18; Pilot Informetlon Plot For Scenario iS*2, Subject il*57 





Ot h 6 R 






Response: Vacuum Pump Failure 

Confidence in Response: 5 

Criticality; 4 

Total Flight Time: 301-500 Hours 

Total Diagnosis Time: 241 Sec. 

Rating; Private 

Key Term ; N 

Total No. Inquiries: 25 

Knowledge Score; 50% 
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Figure 3-19: Pilot Information Plot For 
Scenario <il3, Subject l)*57 
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Schama Analysis 


For purposes of comparison, schema diagrams were prepared 
for two groups of subjects for each diagnostic scenario. The 
"correct” subjects were those who scored the maximum of five 
points on a given scenario while the "Incorrect" ones scored 
zero. Each diagram presents the percentage of subjects who 
included the Indicated Information In their search versus the 
median percent order In which that data was requested. (See 
Figures 3-21 through 3-29). 

The schema diagrams for correct and Incorrect groups In 
scenario 1 (Figures 3-22 and 3-23) look remarkably similar. 

This scenario Involves a broken oil pressure gauge line behind 

i 

Che Instrument panel. Both groups show a high percentage of 
interest In oil pressure, oil temperature, and cylinder head 
temperature (cluster 1) very early In the search process. They 
also have a similar Interest In external evidence of oil on the 
cowling and windscreen with roughly half of the subjects making 
such Inquiries (cluster 2). General engine health as evidenced 
by cluster 3 seems Co be of less interest. The only distinguishing 
characteristics appear to be that the correct group looked for fluid 
leaks, which was the key element, and was concerned with Inside 
physical evidence such as housekeeping and cabin temperature 
(cluster 4) while the Incorrect group ignored this information. 

* Scenario 2 (Figures 3-24 and 3-25) shows more variation 
between groups. The only common element shows up in cluster 2 
which concerns Information related to Icing conditions. Roughly 
half of all subjects seem to believe that this problem, a vacuum 
pump failure, is Ice related. The successful group does check 


cluster the gyro Instruments and suction gauge, while very 
few In the unsuccessful group bother to look. Cluster 3 in 
the correct group represents a small minority of inquiries 
concerning general aircraft health. The incorrect group does 
not evidence any strong clustering tendency beyond their con- 
cern for ice. 

The distinction between correct and Incorrect groups in 
scenario 3 appears to hinge on "cluster fixation" (Figures 
3-26 and 3-27). Both groups show a timely concern for general 
engine health, cluster 2, but only the correct group breaks 
out of the general health symptoms to check the engine ignition 
system. This cluster (H) of course contains the key element 
since the symptoms are caused by a broken right magneto drive 
gear. 

Scenario 4 Involving a frozen static port is somewhat perplex- 
ing in view of the great Interest in icing symptoms evidenced 
throughout all the scenarios (Figures 3-28 and 3-29). Although 
both groups do show strong Interest in the potential for ice, (cluster //I) 
the incorrect group seeks that information early and then abandons 
that track before reaching the key element, The correct group 
shows a much broader concern and keeps hammering at the ice question 
until they uncover the static system problem, albeit rather late 
in the search process. 


Figure 3-21 . LIST OF SYMBOLS USED IN SCHEMATA, 


Abbreviations 


Aileron Gond • AC 
AirmeCs - Air 
Alt. Static ■ A/S 
Alt. Static Closed ■ A/SC 
Alt. Static Open ■ A/SO 
Breaker Panel - BP 
Cabin Temp ■ CT 
Cargo ■ Car 
Ceiling ■ Ceil 
Cowling Condition * CC 
Cloud Tops ■ CLT 
Cycle Head Temp « CHT 
Flap Condition « FC 
Flaps ■ F 
Fluid Leaks ■ FL 
Freezing Level ■ Fr 
Fuel Selec » FS 
Fuel Selec. Left • FSL 
Fuel Selec. Right * FSR 
Ground Speed ■ GrS 
Left Fuel Quant. ■ LFQ 
Housekeeping » H/K 
Mag Both « Mag B 
Mag Left ■ Mag L 
Mag Right » Mag R 
Master Switch *» MS 
Master Switch On * MSO 
Master Switch Off = MSOF 


Mix ■ Mx 

Mix Enrich ■ MxE 
Mix Lean ■ MxL 
Nav. Aid Stat. ■ NAV 
Noise & Vibration - NV 
Oil Pressure ■ OP 
Oil Temp • OT 
Panel Temp ■ PT 
Plreps - Pir 
Pitot Heat - PH 
Pitot Heat Off - PHOF 
Pitot Heat On - PHO 
Prop RPM - PR 
Prop PDM Deer ■ PRD 
Prop RPM Inc » PRI 
Right' Fuel Qty - RFQ 
Slgmets “ Slg 
Smoke « Sm 
Stabilizer Cond ■ SC 
Suction ■ Sue 
Throttle - Thr 
Throttle Dec - ThrD 
Throttle Incr * Thrl 
Winds Aloft =» WA 
Windscreen Cond ■ WdC 
Wind Cord - WC 
Visibility - Vis 
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Subjects: 42, 43, 4e, 47. 51. 52. 53. 54, 55. 56. 57, 59, 60, 61, 64, 67. 68. 80 


Frequency (%) vs. Order (Median) 



Order (%) (Median) 

Figure 3-22: Schema For Subjects In Scenario //I With Correct Diagnosis 

(See Figure 3-21 for the list of symbols used in schemata) 



Subjtctss 45, 48, 49, 50, 58, 65 
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Frequency (%) vs. Order (Median) 




Order (%) (Median) 

Figure 3-23: Schema For Subjects In 

Scenario #1 With Incorrect Diagnosis 


ORIGU^AL PAGE B 
OF POOR QUALITY 


- 92 - 


Subjects: 45» 49, 50. 51, 57, 63, 71. 72, 77, 80 


Frequency (%) vs. Order (Median) 



Order (%) (Median) 

Figure 3-24; Schema For Subjects In 
Scenario //2 With Correct Diagnosis 
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Subjeoti; 42, 44, 46, 48, 52, 54, 55, 56, S8, 59. 60, 62, 64, 65, 66 
66, 68, 70, 73, 75, 76, 78, 79, 81, 82, 83, 84 


original page is 
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frequency (%) ve. Order (Median) 



Order (2) (Median) 

Figure 3-2 S: Schema For Subjects In 

Scenario #2 With Incorrect Diagnosis 
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i 

1 

Subjects! 42, 43, 46, 49, 50, 51. 52, 53, 54, 56, 58, 59, 60, 63, 65, 
71, 73, 75, 78, 80, 82, 83 


Frequency (X) vs, Order (Median) 



Order (%) (Median) 


Figure 3-2 6s Schema For Subjects In 
Scenario #3 With Correct Diagnosis 
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SubjectiJ 45, 47, 48, 55, 56, 61, 62, 66, 68, 72 


Frequency (%) ve. Order (Median) 



Order (%) (Median) 


Figure 3-27; Schema For Subjects In Scenario //3 With Incorrect Diagnosis 
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Subjects: ^2, 43, 46, 48, 50, 51, 53, 55, 58, 61, 64, 65, 70, 7i, 78 

i 

I 


Fre<tuency (%) vs. Order (Median) 



Order (%) (Median) 


Figure 3-28: Schema For Subjects In Scenario #4 With Correct Diagnosis 
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Subjects; 45, 47, 49, 54, 71 


Frequency (X) vs. Order (Median) 




Order (%) (Median) 

Figure 3-29; Schema For Subjects In Scenario H With Incorrect Diagnosis 
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G. D««tlriatlon Dlvcyglon Perfonnance 


The destination diversion problem Involved selecting an 
airport from among sixteen candidates d;(strlbuted around the 
geographic area In which the alternator failure occurred * 

Six Items of Information could be requested for each airport 
during the two minute search period. The Information avail- 
able Included: (1) bearing and distance, (2) celling, (3) vis- 

ibility, (4) approach aids, (5) ATC services, and (6) terrain. 

The location of the airports relative to the point of failure 
la depicted In Figure 3-30. The problem is somewhat complicated 
by the presence of a strong southwest wind which was noted In the 
presentation of the scenario. 

The characteristics of each of the sixteen airports are 
noted In Table 3-10. The characteristics were chosen to represent 
extremes in desirability, e.g. mountainous Versus level terrain, 

ILS versus NDB approaches, 500 foot versus 1000 foot ceilings, etc. 

The number of information requests among the thirty subjects 
who completed this part of the experiment Is summarized in Table 3-11. 
The items of information most often requested were ceiling and 

I 

visibility which seems consistent with the order of information 
in weather reports. The low number of requests for terrain informa- 
tion (rank 5 of the 6 classes) is somewhat surprising. One wonders 
about the wisdom of selecting an airport on the basis of electronic 
aids which may face imminent failure without knowing the potential 
hostility of surrounding terrain. The airports receiving the most 
information requests were airports 3 and 5. The high frequency of 
requests for those airports may be due simply to their perceived 
proximity to the point of failure as determined from the simplified 
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[ Rpproach 
1 Rids: 

- H 


RTC 



Services: 
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* . 

Terrain: 
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SELECT RIRPORT 

GIVE 

Ml RNSUiER 


B<sar i ng ^ -- X 

Distance: 

Cei ling; — 
Visibility: “’3 

SUI1MRRY INFORflRTION 


Figure 3-30. SIMPLIFIED ENROUTE CHART FOR DIVISION DECISION 







Table 3~10; AIRPORT CHARACTERISTICS 
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enroute chart. However, had the aubjecta conaldered wind 
effecta, airport 3 ahould have received much leaa attention 
than aome other a. 

The most popular choices for destination diversion alr» 
ports were airports 1 and 3. On the surface these two represent 
distinctly different choices. Both are equidistant from the 
point of failure. However, airport 1 la In level terrain, down- 
wind and has an ILS approach, while airport 3 Is In hilly terrain, 
upwind with a non-preclalon VOR approach. Celling and visibility 
are SOO and 1 for airport 1 versus 1000 and 3 for airport 3. 
Several t-fiests were run to test the effect of flying experience, 
knowledge test scores, diagnostic scenario performance and airport 
Information type requests on the choice of one of these two 
airports. None of these provecl to be statistically significant. 
Whatever Influenced the choice of airport 1 versus airport 3, 

It Is not Immediately apparent from the subject characteristics 
examined . 

Figures 3-31 through 3-33 present relative frequency informa- 
tion derived from the table of information requests. Here it is 
possible to see at a glance the dominance of airports 1 and 3 
and information on celling and visibility. It Is Interesting to 
note that the information frequency profile for those two airports 
Is virtually identical. Profiles for other airports exhibit 
decidedly different patterns. 




Figure 3-31: Relative Frequency of Selection of Each Airport 




- 105 - 

Plgur« 3-33: Relative Frequency Of 

Information Roqueats By Airport 

ICey: 1-Bearlng & Dlatance 

2-Celllng 

ORtGtNAL PAGF fo 3-Vlalblllty 

POOR OUAl/r? 4-Approach Alda 

VU«l»TY 5_ATC Facllltlea 

6-Terraln 


Airport #l 
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.15 - 

RELATIVE 

FREQUENCY .10 . 

.05 - 

0 -U 

1 2 

INFORMATION 

Airport #2 - No Information Requested 




Airport ^3 


RELATIVE 

FREQUENCY 



Airport #4 


RELATIVE 

FREQUENCY 



INFOR.MATION 


Flgur* 3-33 (conclnuai) 
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Airport #5 


Kay: 1-Baarlng & Dlstanca 

2- Calllng 

3- Viaibillcy 

4- Approach Aids 

5- ATC Pacllltlas 

6- Tarraln 
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Figure 3-33 (continued) 
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Al-oort #U 


RELATIVE 

FREQUENCY 


Key: 1-Bear Ing 4 Dlatance 
2-Celllni 



INFORMATION 


Airport #15 


RELATIVE 

FREQUENCY 



Airport #16 


RELATIVE 

FREQUENCY 



INFORMATION 
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A NtAtlielciil protiiLd of th@ time and Inquiry patterns 
of subiecta is contained in Table 3«12* The items of note here 
are the wide variations in number of inquiries per subject and 
number of airports considered, A aeries of t-testa were tun 
to distinguish the behavior of the professional versus the non- 
profeasionel pilot. The only atatistically significant result 
was that non-pro pilots required a longer average time to choose 
an airport once their data gathering was complete, 


ITEM 

MEAN 

RANGE 

VARIANCE 

Inquiries 
l?er Subject 

22 

6-44 

84.8571 

Airports 
Per Subject 

6.28 

2-lS 

9,2069 

Information 
Per Airport 

3.82 

2-6 

1.4292 

Time 

Per Inquiry 

3*44 sec. 

2.15-5.88 

1.1442 

Time To 
Choose Airport 

14.02 sec. 

6-26.5 

21.7067 

Time 

per Airport 

27.39 sec. 

13,11-52.33 

89,2388 


Note: Data bused on performance of 29 subjects. 

Table 3-12, Destination Diversion Time and Inquiry Statistics 

Individual performance differences are best noted in the 
destination information graphs, Figures 3-34, 3-^35, 3-36 1 and 3-37 depict 
four subjects with different information needs and information seeking 
styles. Figure 3-34 shows a subject who collects all the information 
available from each airport selected. Figure 3-3S depicts a subject 
who baa concern for celling and visibility but not much consciousness 
about terrain and qthar attributes. Figure 3-36 depicts a subject who 



chooses an airport as soon as it meets his criteria and £alls 
to use available time to locate "better" diversion airports. 
Figure 3>*37 depicts a subject who first fOcv^ses on approach 
aids and only seeks added data on those meeting his approach 
aid criteria. The secondary Information he seeks Includes 
celling and visibility. 

These DIGs are merely representative of the general styles 
of information search exhibited. Although each subject can be 
categorized Into one such pattern, there Is no discernible 
relationship between the Way subjects collect Information for 
diagnosis as Illustrated by PIFs and the way they collect Informa 
tion In destination diversion decisions as depicted by DIGs. 
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Figure, 3-34: Destinatlon-Oiversion Total Information Search 
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Subject i^7B 

Time Lapsed; 189 sec. 

Sec. /Port Selection; 10.55 
Sec. /Inquiry ; 3.0417 

Airport Selected: #3 

Mote Mos. in the box refer to a specific airport 



Figure 3’-35: Destination Diversion Limited Information Focus 



Subject #70 

Time Lapsed: 78 sec. 

Sec. /Port Selection: 9 

Sec . /Inquiry : 5.45 

Airport Selected: #1 

Note Nos. in the box refer to a specific airport 




FREQUENCY OF ATTRIBUTE 
INFORMATION SOUGHT 



Figure 3-36: 


Destination Diversion Quick Decision 
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. 1 


Subject: 

Time Lapsed: 129 sec. 

Sec. /Port Selection: 9.625 

Sec. /Inquiry: 3.25 

Airport Selected: //3 
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Note Nos. in the box refer to a specific airport 



7 

(9 

5 

FREQUENCY OF ATTRIBUTE*! 
INFORMATION SOUGHT 

I 

I 



ATTRIBUTE INFORMATION 


Figure 3-37: Destination Diversion Primary Criterion Search 
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H. An Experiment Combining Diagnosis and Destination Dlveralon 
Dtcislona Within Computer Aided Testing 

Task 6 of the research plan called for an experiment combining 
the diagnostic and destination decisions within the computer 
aided te^ts. This required a simulated full mission scenario which 
embodied not only a CIFE but also navigational and communication 
inputs from the subject pilot. This necessitated rather sophis- 
ticated programs for the PLATO® system. For navigational responses 
the aircraft had to be located in space using VOR fixes and the 
computer had to account for heading and altitude changes imparted 
by the pilot through an auto pilot display. The navigational and 
autopilot display is shown in Figure 3-38. The pilot was given a 
simulated low altitude chart to locate his position relative to 
VOR’ 8 and available diversion airports (see Figure 3-39). 

Coiranunlcatlon using CAT is difficult because it limits the 
number of messages the pilot can communicate if one wishes to keep 
a menu driven display system and not require complex typed responses 
by the subject pilot. 

Communication from ATC to the pilot is less a problem because 
alpha numeric messages to the pilot are relatively easy to present 
given that the computer recognizes the pilot’s position within the 
flight. Figure 3-40 depicts the means by which the pilot communicates 
with ATC. This display is more involved than the ATC information 
display used in the simple diagnostic tests. The display permits 
the pilot to ask for information and also to give position reports 
and declare emergencies or request special handling. 
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Figure 3-38; VOR-AUTO PILOT DISPLAY 



















Figure 3-39: SIMULATED LOW ALTITUDE CHART 
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Pilot would like to advise ATC of a problem and 
mac need to make heading and altitude ckianges . 


Figure 3-40; ATC COMMUNICATION DISPLAY 






original page is 

OF POOR QUALITY 

The scenario used in this experiment Is a modification of 
scenario 5 described In Chapter 2. Essentially , It was decided 
to give the pilot a problem which would not likely be solved 
within the time available. A broken baffle was selected to 
ultimately force the pilot to consider his decisions about 
selection of a destination airport. 

The simulated full mission scenario located the pilot such 
that he was heading Into the mountains with an unresolved problem, 
l.e. unable to get full power from the engine and required by 
ATC to maintain 5000 feet In IMG conditions. Figures 3-41 through 
3-45 depict the typical development of the situation with ATC 
communication to enhance position awareness. Table 3-13 Investi- 
gates some of the Inquiries made by a pilot attempting to resolve 
the problem. 

Results 

Table 3-14 describes the various responses made by pilots using 
this "full mission scenario". The Intent of this study was to 
force a choice dilemma situation, l.e. the pilot could elect to: 

(a) struggle on over the mountains wlti) a sick engine hoping no 
other problems develop, (b) declare an emergency and seek to land 
the aircraft immediately In minimum VFR conditions at a nearby 
private airport, or (c) to return to the airport of origin where 
instrument aids would be available. It was also of Interest to 
ascertain how long the pilot would attempt to diagnose his problem 
before accepting the fact that he would be unable to get full power 
from the engine. 
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I biflGMOSTIC SCEIsIBRIO TEST 1 

(lie are now going to present to you. some 
Critical In-Flight Events I'equiring your 
diagnosis of the problem. 


Bssume that you are flying a fuel- 
injected Cherokee arrow (N123B) with the 
f o 1 1 oi'J i )’ig pel"' f ormance spec i f i cat i oti-s : 


Cru i se Speed 


135 KTfiS (65% pwr. rj.- 70.00 ft.) 


Fuel Flow (6 5% pwr.) 


10 GPH 


Usab I e Fue I Capac i t y = 4 8 ga 1 1 o ns 


Endurance 


4 . 8 I'tours (no r eserve) 


Range = 648 nautical miles, (no wind, no reserve) 


Press CONTINUE when finished reading. 


CONTINUE 


Figure 3-41; SCENARIO INITIALIZATION 
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Ccri5u 1 1 AttAcl.-td low Altituoe civirt. 

Y>:hj *r»t on &n IKR flight from Ut*h Munici|:al f^ir|;*c*rt 
to Hav^sn Ccunty Airport. Y:>u depart on V-110 at 6000ft 
in your Cherokee Hr row (N123B) which is eq'.«i|:<ped 
with a 3 -ay is actopiloc. Tl*.ere is a NOTRM out which 
»'e|:>ort» that Colorado VOR is out of service during 
tl-ke period >'OU p*lan to navigate. Na vij^ate usin^ Ohi^h 
a>~td Cal i f or n i a VCRs . You have been enroute 60 minutes 
from Utah Municipal Airport. You. are on the ^aujes 
but the ride is smooth. lOeatkior briefirig indicated 
that loinds at 6000 toere expected to be light a»>d 
var i able. 

You have one passeriger aboard. 

Weather at: 

Haven County A i import* 2000 It 5 
Ohigh- 1000 & 3 

Wind Falls- 1000 & 3 by a C-172 
(10 minutes ago) 

Cleve Center calls and reports radar contact is lost. 
Please report present position. 



When ready, press the CONTINUE button to go 
to the VOR display to establish position. 


Figure 3-42; SCENARIO SETTING 
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FtKure 3-43: POSITION REPORTING 





ORIGINAL PAOC \S 

I l. aat Cl ear Alice | qP poOR 0^^^* 

hTC 

NlZ'-tB, for th« |>'*«ition r*»port. 

ie your cl <?<?»» one 

yroct^cd dir**^,t Colifortiia V0*R diT'o-t 
Hov**»i County f^irport <>t 6000. 


r)i«ii «5 «*Ji 1 1 l:»c Of.*pos:t«t traffic 
dt !^U00 . . . rno i ntd n I o0i’0. 

co»tf i»m y»:*ur »vno ^l«*»^din‘g 
oittd altitude ofler y*-*ur lu» ri. 



Figure 3 - 44 : 


ATC ReHpon»*o 
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SCEMriRIO CHRI IGE 

While practicing h.and flying 
with your autopilot di eeng.sged, 
you Tiotic« tliat increas-ed noa><s-up 
trim is requ i red to ma i rit a i n a 
constant i indicated altitude arid 
that your IRS has decreaied 2tfkts. 
f r om nor nria 1 cru i se . 

Your frassenoer notes this problem, 
and suggests that you turn back 
to Ut ahi Mun i c i pa 1 . 


Determine the nature of the problem, 
arid your dest i I'lat i on dec i s i on . 



Figure 3-45: INTRODUCTION OF THE CIFE INTO THE SCENARIO 
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3-13; Illustrative Pilot Inquiries 

in the Combined Scenario 
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Early trials Indicated a complete preoccupation with 
diagnosis and little position awareness. Attempts were made 
to Increase position awareness through position reports to 
ATG and a passenger insisting they return to the origination 
airport. Despite these efforts, most pilots indicated little 
concern for altitude clearance violations and spent most of the 
time wrestling with the engine problem. Most opted to continue 
over the mountains without appraising ATC of the problem, 
though some pilots did declare emergencies or return to the 
airport of origin. Despite the fact that some pilots had exper- 
ienced this engine problem earlier in diagnostic tests, there 
was little evidence thsw they recognized this problem or abandoned 
the diagnostic Inquiries earlier than pilots viewing this problem 
for the first time. Table 3-14 shows that there is a considerable 
variation in diagnosis time, number of • inquiries , and return to old 
track inquiries (DIPT) . Only one pilot declared an emergency and 
none elected to put the aircraft down at the nearby private strip. 
Table 3-15 reflects performance differences of subjects Who took 
scenario 5 as a pure diagnosis problem as compared to those who 
took it in conjunction with the destination diversion decision 
(denoted as P-G on PLATO® GAT) . Again we see a wide variety of 
r^jsponses from both groups. The P-G group took more time on the 
first track and made more inquiries in general. This probably 
reflects the stress of the navigation decision. 

The general conclusions from this experiment were: 

(a) using CAT it is possible to provide realistic displays 
for navigation purposes; 
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Table 3-14 Sunmary of Performance In Combined Scenario 











































































































































Table 3-15 T 


In Scenario 5 Alone and in TlaCo-GaC 


Subject; // 

Seen 5 
or 

Plato Gat 

First 

TracR 

Inq. on 
1st Track 

Time on 
1st Track 

2nd 

Track 


56 IFR 

5 

Struct. 

Fail. 

2 

27 

Other 


57 IFR 

5 

Other 

(Suction) 

1 

13 

Ice 


58 IFR 

5 

int. Enq. 
(MP) 

1 

. J 1 16 

Ice 


59 IFR 

5 

Ice 

1 

16 

Flight 

Status 


60 VFR 

5 

Other 

(CatfiO Cbnd) 

1 

27 

Struct. 

Fail. 


61 VFR 

5 

loe 

4 

31 

Other 

(celling) 


62 VFR 

5 

Other 

(Cargo Oond) 

2 

41 

Ice 


63 VFR 

3 


2 

46 

Struct. 
Fail . 

64 VFR 

5 

Ice 

2 

26 

Prop 

(TACH) 

< 

65 VFR 

5 

lilt. Eng. 

3 

19 

Fuel 

66 VFR 

5 

Ice 

3 

26 

Other 

(grounds ne’ed) 

67 VFR 

5 


1 

11 

Prop 

(N&V) 

68 VFR 

5 

Other 

(Suction) 

1 

7 

Prop 

(TACH) 

69 IFR 

5 

■HU 

1 

. ^ 

Ice 

70 IFR 

5 

lESSHI 

0 

M 

59 

Fuel 

(Throt.) 

76 VFR 

PG 

Int. Eng. 

5 

49 

Ice 

77 VFR 

PG 

Struct. 

Fail. 

3 

« 

44 

Ice 

78 IFR 
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(b) most pilots are reluctant to accept the fad 
that no diagnosis or solution Is available In 
flight. This may result from the fact that all 
other scenarios presented had definitive solutions; 

(c) diagnosis occupied the pilot's time much more than 
positional awareness; 

(d) considerable variation In pilot response to this 
"full mission scenario" was apparent; 

(e) no clear relationship between pilot Information seeking 
and decision making and pilot experience and training 
was noted (probably due to the small sample sizes) . 

In effect, this experiment attenpted to produce a GAT simulator 
experience using a computer terminal. Upon subject debriefing It 
was apparent the pilots were overloaded with Information and 
reacted to the problem by treating the engine symptoms and 
navigation problems Independently as opposed to an Integration of 
diagnosis and diversion information. In their defense the subjects 
argued that in a real aircraft, positional (navigational) and 
engine information would be simultaneously available. They also 
argued that a one time presentation of this kind of scenario gave 
them little chance to adapt, l.e. they had little chance to do 
VOR tracking prior to the CIFE. Unfortunately, real ClFE's give 
pilots little chance to adapt. 

There was Insufficient time or funding available in the research 
program to pursue this "full mission scenario" using CAT beyond 
these exploratory experiments. It may be that trying to produce 
a type of CAT-1 simulator using computet dis'.lays Is not an 
efficient or realistic approach. 

Despite the question of whether this approach has value in 
understanding pilot logic structures, there appears to be considerable 
potential for pilot training. 


I . CoB!iparl«on of Computer Aided Testing With Paper and Potncll 

The four basic diagnostic scenarios and the destination 
diversion scenario used In computer aided testing (CAT) were 
baaed on the same underlying problems as those used In earlier 
paper and pencil (P/P) experiments. The principal differences 
Involved manner of presentation and timing. CAT was nearly 
experimenter free. Subjects received Instructions and sought 
Information from an interactive computer graphics display. The 
computer recorded the sequence of Information exchanges and the 
time between Inquiries. The P/P tests on the other hand required 
a verbal exchange between experimenter and subject to transfer infor- 
mation. Although the experimenter did record the order of Inquiries, 
there was no provision for obtaining a complete time history. 

Subject Characteristics 

The sample sizes for CAT and P/P testing were nearly identical. 

CAT used 42 pilot subjects while P/P used 40. A quick comparison 
of the relative flying experience of the two groups can be seen 
in Figure 3-47, and 3-48. Their relative performance on the know- 

ledge test is shown in Figure 3-^9. Both groups had identical 
questions on the knowledge test instrument. 

The general impression is that the GAT test group is less 
experienced. The CAT group has more private pilots, lower total 
flying time and lower single engine time than the P/P test group. 

The entire group of P/P subjects hold the Instrument rating while 
sixteen pilots in the CAT group are not instrument rated. The P/P 
group scored slightly higher in the knowledge test. These character- 
istics are summarized in Table 3-16. 
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Figure 3-46: Certificates Held By Subjects, P/P versus CAT 
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Figure 3-47: Total Flying Time By Subjects, P/P versus CAT 
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Figure 3-48: Single Engine Flying Time By Subjects, P/P versus CAT 




Figure 3-49: Knowledge Scores By Subjects, P/P versus 
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SUBJECTS 

PVT 

RATINGS (X) 

com 

ATP 

AVG 

TT 

AVG 

SET 

KNOWLEDGE 
TEST (%) 

CAT 

42 

42 

42 

16 

1654 

933 

57 

P/P 

40 

22 

48 

30 

3821 

1911 

62 


Table 3-16. CAT and P/P Group CharacCerlatlca 


Diagnostic Performance 

Correctness scores for each of the four basic scenarios were 
based on five points for a perfect answer. However, the grading 
mechanism did change slightly between the P/P and CAT experiments. 

For that reason the relative differences among scenarios may be more 
significant than raw score differences between test groups. The 
average scores are depicted in Table 3-17. It should be noted that 
all P/P subjects took every scenario while not all of the CAT subjects 
did. Consequently, adding average scenario scores for CAT will not 
yield the total average shown. 


Average Scenarios Correctness Scores 



TOTAL CORRECT 

SCENARIO 1 

SCENARIO 2 

SCENARIO 3 

SCENARIO 4 

CAT 

47 

3.3 

1.5 

3.3 

3.0 

P/P 

65 

2.7 

2.9 

3.4 

4.0 


Table 3-17. Correctness Score Comparison 
The higher average score for the P/P group is indicative of 
the generally higher level of flying experience of that group 
compared to the CAT group. An interesting finding occurs in the 
relative rank of scenario 2 scores. Scenario 2 involves a vacuum 
pump failure and subsequent loss of gyro instruments. The P/P group 
scored relatively higher on that scenario than did their CAT counter- 
parts. Since the CAT group contains a number of non-instrument rated 
pilots, this result may suggest that pilots who must routinely 



deptnd on their gyroe ere more cognizant of unuiual Indlcatlone. 
Table 3-18 comparea general aearch pattame between the two 
experiments. The Interesting thing here Is that, with the excep- 
tion of scenario 1, the P/P group made far fewer inquiries per 
scenario than did the CAT group. As one might expect, the CAT 
group also explored a higher number of tracks. This may auggest 
that the computer aided testing is more "user friendly" and Invites 
a broader search for alternatives than does the verbal exchange 
In a paper and pencil format. One might also speculate that 
computer users are leas inhibited about asking for what may be 
frivolous information than are their verbal counterparts who risk 
sounding foolish to the human experimenter who must respond to 
their questions. 

Regression analyses performed on CAT scores reveal that know- 
ledge subscores and source of flight training are predictors for 
correctness scores on some scenarios. These results are consistent 
with regression analyses performed on the P/P data. The only 
striking difference between the groups is that correctness scores 
by scenario seem to be related for selected pairs among CAT sub- 
jects while in no case is a good score on scenario 1 a predictor 
of score on scenario j in the case of the P/P subjects. 

When subjects are split into high and low groups for t-test 
comparisons, results of CAT and P/P tests are similar. For example, 
in both cases subjects with high knowledge test scores tended to 
achieve high correctness scores and they were also the pilots 
with the most single engine flying time. Since most of these tests 
are based on the same biographical data and knowledge test scores, 
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CAT 


TOTINQl 

6.12 

6.38 

TOTRAKl 

3.33 

1.80 

UNTRAKl 

2.57 

1,50 

TOTINQ2 

15.21 

8.38 

TOTRAK2 

6.76 

3.75 

UNTRAK2 

3.57 

2.75 

TOTINQ3 

15.10 

9.58 

T0TRAK3 

7.14 

3.85 

UNTRAK3 

4.00 

3.15 

TOTINQ4 

13.90 

9.12 

TOTRAK4 

5.76 

4.10 

UNTRAK4 

3.34 

3. 35 


Table 3-18: DIAGNOSTIC INFORMATION SEARCH MEAN VALUES 



-137 

th« lack o£ glaring di££erancaa o££ara evidence that CAT end 
P/P teatlng are almllar In performance meaaurea expected from 
any given claee of aubjecta. 

Destination Dlverelon Performance 

The difference In the mechanics of Information exchange 
between CAT and P/P testing Is again reflected in the Inquiry 
staclatlca for the destination diversion scenario. The average 
number of inquiries Is 22 for CAT versus 17.2 for P/P subjects. 

The information requests per airport shows a similar trend with 
averages of 3.82 versus 2.60 for CAT and P/P, respectively. The 
interactive terminal either Invites a wider search or ntakes the 
information exchange more efficient thus permitting more inquiries 
in a fixed test period. 

The general pattern of airport consideration between the 
groups is similar in that the frequency of consideration for a 
given airport decreases for airports further away. Both CAT and 
P/P subjects showed heavy interest in airport 5. However, the 
final choice of a destination diversion airport is completely different. 

The highest ranking airport among the ?/P subjects was airport 14 
while the CAT subjects preferred airport 3. Both choices have 
similar facilities (i.e. VOR approach and a tower) and both are In 
Che same general direction. They differ in that 3 is in hilly 
terrain while 14 is in the mountains. The final choice may have 
been more influenced by the form of the map than by any real difference 
in preference between the groups. 

The relative frequencies of type of information requested 
are quite similar for both groups. Ceiling, visibility and approach 



aids are the moat often requested information items. The medium 
of presentation does not generally appear to affect preferences 
for type of data in the destination diversion problem. There is 
one uneiiiplained anomaly, however. Bearing and distance informa*' 
tion represents 14.4 percent of the total information request! 
from CAT subjects and only eight percent from P/P subjects. It 
is not clear why this data should be a sizeable portion for either 
group since it can be at. least roughly estimated simply by glancing 
at the enroute chart without wasting time on an information query. 



chapter IV: Learning In Computer Aided Testing o£ CIFE Diagnosis 

Task 7 of the research was directed to subjects learning in 
terms of performance changes both within test sessions and across 
test sessions. The significance of this task can be better 
appreciated when one considers the long range potential of Incor^ 
poratlng CAT in pilot training. Self-paced training would enable 
pilots to proceed through scenarios of Increasing difficulty until 
some level of competency was accomplished. 

Because of the novelty effect in using touch CRT's there was 
a need to see if pilot performance in terms of correct diagnosis, 
efficiency In arriving at diagnosis, and information seeking style 
would change within a test session and across test sessions. 
Insufficient time prevented across tost learning experiments. 

Within Session Learning 

In order to test for order effects and not confound scenario 
difficulty with crder, two order sequences were used. One group 
of subjects received scenarios 1, 2, 3, and 4, in that order. A 
second group were tested in the order 2, 3, 4, 1. Measures of 
performance included: 

(a) the total number of inquiries 

(b) the total number of tracks 

(c) correctness score 

2 

(d) the time interval between inquiries (mean and a ) 

(e) the number of unique tracks 

(f) the total time to complete the diagnosis 

In order to evaluate aggregate learning on performance scores 
it was first necessary to normalize subject data. Data for each 
subject was normalized by taking the average score across the scenarios 
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foe each subject and then dividing individual performance measure 
by the average to get percentages. For example » for a given 
subject, correctness score on a given scenario would be some 
percentage of his average correctness score on all scenarios. 

This technique enables subject data to be added without giving 
heavy weighting to Subjects with large numerical scores. 

Table 4-1 shows a two way ANOVA - scenario vs. order for 
three scenarios (2, 3* and 4) and two orders (1,2»3,4» vs. 2,3»4,1). 
Note that for most performance measures the order of testing had 
no effect. As might be expected, scenarios were somewhat different 
in terms of correctness, number of Inquiries, number of tracks, and 
number of unique tracks. It was generally found that scenario //2, 
the vacuum problem was the most difficult scenario for the subjects. 

Wlien time data is examined, means of time between Inquiries 
(delta time) show a pronounced trial effect, l.e. the first trial 
(for both orders) is higher than all the remaining trials. Figure 
4-1 separates IFR and VFR pilots for each order and depicts changes 
in mean delta times over the trials. The reason IFR and VFR pilots 
were separated was because of possible confounding effects. VFR 
pilots arfe over-represented in the order 2-3-4-1. With the IFR-VFR 
break out it is still clear that learning takes place across trials 
within each order in terms of time between inquiries. Beyond that 
finding, little learning is evident across trials within an order. 

This is an Interesting and reasonable finding. Reduction in delta 
times comes with familiarity in using PLATOtP. Diagnostic performance 
should be dictated by scenario difficulty and not by order of testing. 
From Figure 4-1 it appears that learning in terms of delta time 
would continue beyond the four scenarios tested. 



? 

Table 4-1 Order & Scenario Effects on Performance 






























To test for s confounding of rstlng with order, contingency 
tests were executed In a two by two design (two orders and two 
ratings - VFR/XFR) for the various performance measures. In all 
cases Che hypotheses of Independence were rejected at the ten 
percent significance level. 


Ghaptei;' V: Computer Aided Programming 

A wide variety of information search patterns were exhibited 
by subjects during CIFE diagnoses, some of which were more logical 
and efficient than others. In an attempt to streamline search 
patterns, preliminary investigation of the use of a diagnostic 
computer aid was carried Out. A prototype Decision Support System 
(DSS) to aid a pilot in his decision making during ah emergency 
situation was designed. The suggested model, called the Airplane 
Condition Evaluation (ACE), was patterned after MYCIN, a diagnostic 
computer aid used by physicians in their diagnosis of blood diseases. 

ACE uses an interactive program similar to MYCIN which utilizes 
production rules and allows the user to input the current facts of 
the situation. Each of these production rules contains a small 
amount of Information for ACE to draw upon. ACE is able to "reason" 
with current facts and to "deduce" a conclusion by Joining together 
a series of these production rules. The production rules are modular 
and may be joined together in any way that the current facts support, 
thus allowing ACE to generate a diagnosis with any combination of 
aircraft symptoms. This system is described in complete detail in 
the MS thesis by Jeffrey A. Lee, A Decision Support System For In-Flight 
Emergencies: ACE (5). Mr. Lee is a Graduate Research Associate on 
the NAG 2-112 research project. A copy of that thesis has been 
submitted along with this report and is available at NASA Ames. 

ACE is a "suggestion model" type of DSS which plays the role 
of a consultant with the expertise of an experienced aircraft pilot. 

It has the capability of accepting many variations of the problem 
it is geared for. Since it is computer driven, it has the advantage 


-144 



-145 


of speed which makes it particularly well suited for a real-time 
environment. Such a system could be physically located In the 
cockpit of more sophisticated aircraft or located on the ground 
for use by smaller aircraft. In the ground mode some means 
of data exchange, perhaps as simple as voice radio communication, 
would be required. If the system was located at a Flight Service 
Station, for example, any flight service specialist could serve 
as a surrogate for the expert consultants currently available to 
air carrier and military. ACE would guide the questioning which 
the FSS specialist could relay to the pilot. 

In Its current state of development ACE contains a total of 
fifty-one production rules. These rules were distilled from a 
frequency count of Information requested by PLATO® subjects who 
correctly diagnosed scenario problems and augmented by discussion 
with local "expert" pilots. The experts were used to establish the 
reason why information items requested with high frequency led to 
a correct diagnosis. The experts were able to suggest what could 
be Inferred from a group of Items of Information and also what 
additional Information would be needed to further support the line 
of reasoning. This analysis gave way to the formation of production 
rules and thereby established the expert knowledge and reasoning 
necessary to draw the correct diagnosis for a CIFE. 

ACE has so far been constrained to the four scenarios developed 
In the earlier paper and pencil studies and later extended to the 
PLATO® tests. ACE does not yet have the ability to work in the 
real world but only in conjunction with this set of four scenarios 
and the corresponding set of information for each as presented by 


the PLATOO data-gatherlng computer system developed for this 
research. Within those constraints » however » the 51 production 
rules contain enough expertise to correctly diagnose each of 
the four scenarios. 

The following sample session with ACE Illustrates a typical 
set of computer queries and; subject responses. In testing ACE, 
responses were made consistent with data the subject obtained by 
querying the PLATO® representation of current aircraft status. 

Thus, two computer systems were used, a DEC 20 to house ACE and the 
CDC host computer which houses PLATO®. The accompanying logic trace 
shows the trees used In chaining to the recommended action or diag- 


nosis. 
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Th« folloMlnd is » semr-'ltf session with ACE* 


Welcome to ACE* *, the Airplsne Condition Evslustor 


ACE will try to die^nose a Critical In-Fliiaht Event bw 
askintf «o»j a series of True/Felse cues t ions. Please 


with a *T' 


for e true 
You 


answer each of the ouestibr 

statement or en *P* for a false statement* You maw 
also ask ACE whw it is askina you a particular Question 
by entering 'WHY** ACE will respond with the production 
rule thet it is currently considering to create 
a probable diasnosis of the CIFE. 


Please enter *T* to start the Questioning. 
T 


SUCTION LOU? (T/F> 
WHY 

HYP0THESIS<1> 

if: vacuum problem 

OBU; SUCTION LOW 
ACT! VACUUM FAILURE 


SUCTION LOW? (T/F) 
T 
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AS NOT CONSTANT? (T/F) 

T 

ALT NOT CONSTANl? (T/F) 

T 

<VSI NEGATIVE OR VSI POSITIVE)? (T/F) 
T 

AH LEVEL? (T/F) 

T 

DG CONSTANT? (T/F) 

T 

T/F SHOWING SANK? (T/F) 

T 

AIRPLANE CRUISING? (T/F) 

T 

AS INCREASING? (T/F) 

T 

ALT DECREASING? (T/F) 

T 


THE COMPUTED DIAGNOSIS IS THE FOLLOWING: 
-> VACUUM FAILURE 


^ ^ ^ ^ ^ ^ ^ ^ 4 * ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ 

^ ^ ^ ^ VR ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ n» ^ N* ^ ^ ^ ^ ^ ^ ^ 

DESIRE TRACE OF LOGIC? (T/F) 

T 

DESIRE TERSE TRACE? (T/F) 

T 

TREE<1> 
if: STOP 

OBV: AIRPLANE CRUISING 
OBV: AS INCREASING 
OBV: ALT DEGREASING 
OBV: AH LEVEL 

act: POSSIBLE VACUUM PROBLEM 
TR£E<13> 

if: POSSIBLE VACUUM PROBLEM 
QBV: AS NOT CONSTANT 
OBV: ALT NOT CONSTANT 
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OPV: (VSI NEGATIVE OR VSX POSITIVE) 

Opv: AH LEVEL 

OE<V: HG CONSTANT 

OE«vi t/£< showing PANK 

act: vacuum froplem 

HYF0THE£IS<1> 

if: vacuum froplem 

OPV: SUCTION LOU 

act: vacuum failure 

DESIRE TRACE OF LOGIC? <T/F) 

F 

ANOTHER CIFE? <T/F) 

F 


ACE’s facility for showing the current rule that It Is 
considering and Its ability to show a trace of the rules It 
used to compute a diagnosis make the system more user friendly. 

A user sees not only where a current request for Information 
came from but also may be reminded of possibilities that might 
not have occurred to him. 

ACE Is a prototype of a computer aid that may be useful In 
actual In-f light emergencies. It demonstrates that there Is 
potential In the Idea that a diagnostic computer aid can be built 
with production rules and be flexible enough to handle different 
scenarios and their individual sets of information. 


Chapter VI: Siininary and Concluslona 

The tasks set forth In the proposal on the *'Use of Computer- 
Aided Testing In the Investigation of Pilot Response to Critical 
In-Flight Events" have been accomplished* This project continued 
the early concern for the dynamics of CIFE’s explored in the first 
report for NAS 2-10047 (6) , 

The products of the current research are 1) the First Symposium 
on Aviation Psychology held at Ohio State in 1981 (4); 2) a set of 
programs Including new scenarios for computer aided testing (CAT) 
of subjects in ClFE’s; 3) a dynamic system for CAT permitting com- 
bined diagnostic and destination diversion decisions (PLAT0®-GAT) ; 

4) a prototype Decision Support System to streamline search patterns 
(ACE); 5) a data base of responses from 40 pilots in the Colunbus 
area (80 total when added to earlier paper and pencil subjects). 
By-products of this research so far have included one M.S. Thesis, 
two journal publications and a conference proceedings. 

A, First Symposium on Aviation Psychology 

The First Symposium on Aviation Psychology was held at Ohio State 
on April 21 and 22, 1981, The symposium was supported by this NASA 
grant. The objective of the symposiim was to "critically examine 
the Impact of high technology on the role, responsibility, authority 
and performance of himan operators in modern aircraft and air traffic 
control systems," (4-1) 

The attendee list for the symposium contained 210 persons from 
across the nation^ many with outstanding credentials in aviation, 
engineering, or aviation psychology. Over 40 papers were presented 
at the symposium and published in the proceedings. In addition, 
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selected ones of these papers were published In the Journal of 
Aviation Space and Environmental Medicine . As a result of these 
efforts a second symposium has been scheduled for April 1983. 

B. Computer Aid Testing Demonstration 

The research team was successful in converting paper and pencil 
scenarios for diagnosis and destination diversion decisions into a 
computer interactive touch CRT system. This system provides the 
following advantages over the earlier paper and pencil studies. 

1) The system is experimenter free, l.e. detailed Instructions 
are given via the graphics display permitting subjects to 
run the system without the physical presence of an 
experimenter. 

2) Subjects are enth('>dlastlc about the preset;tatlon format. 

Their motivation remains high for test pi^rlods of up to 
90 minutes long. 

3) Subject response requires only a simple touch of the CRT 
screen. Keyboard entries are not required after initial 
sign-on. 

4) Menu response foimats reduce ambiguity and facilitate data 
analysis. 

5) A complete catalog of pilot responses is available in the 
data display module. 

6) A complete time history of pilot inquiries and responses 
is recorded. 

* 

7) National testing capability is present without moving 
experimenters or equipment. The many CDC Learning Centers, 
throughout the U.S. as well as privately held PLATO® termi- 
nals can be used to call up and rim the programs on site. 

Data files are maintained centrally. 

From comments made by pilots who participated and from observations 
by the research team, it appears that computer aided testing (CAT) 
offers a realistic method to study pilot response to critical inflight 
events. In addition, CAT has the potential to become a valuable train- 


ing aid. 



C, The PUTQft-GAT System 

The term "PLATC^GAT was coined to describe the dynamic navigation 
and control system developed for computer aided testing. The task to 
be addressed was one of combined destination diversion and diagnosis 
decisions. The PLATOB>-GAT system demonstrates the following capabilities: 

1) The system provides an automatic tracking feature which 
permits the experimenter to place the subject aircraft 
anywhere In the simulated test area, then track the movement 
of the aircraft over time. 

2) The pilot can determine his position in space by cross- 
checking on dual VOR Indicators. The indicators are dynamic 
and show course deviations over time. 

3) The pilot can alter his simulated heading and altitude by 
appropriate touch-panel entry on the auto-pilot display. . 

4) ATC position reports and clearances are part of the scenario. 

5) The system combines in the touch CRT both the destination 
diversion decision and the problem of diagnosis. 

The intent of PLATOS^GAT was to introduce a more realistic frame- 
work for tests which would begin to approach the fidelity of earlier 
GAT simulations. The dynamics associated with the decision making 
portion of a GAT-type simulation experiment were preserved without 
the accompanying stick and rudder manipulation. This was accomplished 
on a simple set of CRT displays. 

D. Computer Aided Prompting 

A prototype decision support system to aid pilots in decision 
making during emergency situations was developed. The model called 
Airplane Condition Evaluation (ACE) was patterned after MYSIN, a 
diagnostic computer aid used by physicians in the diagnosis of blood 
diseases. 

ACE has the following characteristics: 

1) It is an interactive program which permits user input on the 


current facts of a situation. 

2) ACE ’’reasons" with curr^^ir; “-^cts to "deduce" a conclusion 
by Joining together a series of production rules. 

3) Although designed around the four basic CAT scenarios, ACE 
has the capability of accepting many variations of the 
problem It Is geared for. 

4) ACE Is user friendly In that it shows the current rule being 
considered as well as a trace of the rules It used to com- 
plete a diagnosis. 

E. Results from CAT 


A series of correlation studies, stepwise regression analyses 
and t-tests were performed on the combination of pilot background 
variables, knowledge survey results, diagnostic scenario performance 
and decision making measures. Among che observations made from these 
analyses are the following: 

1) Of che four basic diagnostic scenarios tested, pilots have 
the most difficulty in Identifying the symptoms of a vacuim 
pump failure. 

2) Pilots who score well In the knowledge test are good at 
diagnosing vacuum pimp and static system failures. 

3) Knowledge Is inversely related to diagnostic Inquiries, i.e. 
knowledgeable pilots reach conclusions (right or wrong) 
more rapidly than others. 

4) Less experienced pilots tend to use a larger number of 
diagnostic tracks than do more, experienced pilots. 

5) Civilian trained pilots score higher on the vacuum pump 
failure diagnosis than do military trained pilots. 

6) IFR rated airmen receive higher knowledge scores and higher 
diagnostic correctness scores than do VFR rated airmen. 

7) High correctness scores are positively related to high 
mean time between inquiries, i.e. pilots Who take time to 
absorb and manipulate current information before seeking 
new data are more apt to be successful than those who 
rapidly collect large amounts of disjoint information. 

8) The data most often requested in diversion decisions are 
ceiling and visibility. Terrain receives a low number of 
inquiries. 



9) Pilots often neglect winds aloft information in selecting 
an alternate airport. 

10) There is no discernible statistical dl£fference in the time 
or Inquiry performance of professional and non-professional 
pilots in selection of alternate airports. 

In addition to the more formal statistical comparisons, special 

graphical devices were used to analyze pilot performance. These de- 

■ i 

vices were used to analyze pilot performance. These devices were the 
pilot information plots (PIP), schema diagrams, and destination graphs 
(DIG) . Among the observations made from these analyses are the fol- 
lowing : 

1) Pilots follow a wide variety of different search patterns 
during diagnosis. 

2) Pilots tend to exhibit similar search strategies across 
all scenarios, i.e. a pilot with a very systemptic search 
pattern on any one scenario will behave slmila* i, ' on all 
others. 

3) Based on PIP and Schema Analyses, efficient information 
searching can be hypothesized.! It appears that the ideal 
pilot should do the following: 

1) Confirm symptoms. 

li) Establish engine health status. 

Hi) Establish an hypothesis concerning cause. 

iv) Determine hypothesis plausibility with a minimum 
number of inquiries on an appropriate track. 

v) Givai logical cause of the symptoms, test alternative 
hypotheses by additional information Inquiries. 

4) Schema diagrams are scenario dependent. 

i) In the oil leak scenario, the only distinguishing 
feature of correct versus incorrect groups is that 
the correct group focused on internal housekeeping 
questions while the incorrect did not. 

ii) In the vacuum pump failure scenario, the only common 
element between the correct and incorrect groups is 
a strong concern for icing indications. Only the cor- 
rect group bothered to check gyro instruments. 


Ill) In Che broken magneto scenario, Che correct group 

checked Che Ignition system while the Incorrect one 
did not, 

Iv) In the frozen static port scenario, both groups sus- 
pected Ice, but the Incorrect group abandoned that 
track early In the search. 

5) Several general styles of information search strategy can 
be discerned from the DIG’s. 

6) There is no discernible relationship between the way a pilot 
collects Information for diagnosis and the way he collects 
Information for the destination diversion decision. 

The order of scenario presentation was different for a subgroup 
of the CAT subjects. This was done to test for learning effects within 
the diagnostic test session. 

The only discernible learning effect is with respect to time be- 
tween inquiries. The first trial in CAT for both orders of presentation 
has a higher mean time between inquiries than all subsequent trials. 

The reduction in inquiry time apparently comes with familiarity in 
using the touch CRT system. The responses of pilots are essentially 
scenario dependent. Number of inquiries, number of tracks and number 
of inquiries per track all depend more on the content of the scenario 
in question than on its order position within test session. 

Four subjects were re-tested on scenario two (vacuum pump failure) 
several months after their first exposure to that scenario. Correct- 
ness scores and track performance showed little change. However, the 
mean and variance of time between inquiries were lower for the second 
trial. 

Although the data samples are small, the evidence Indicates that 
learning in terms of timing of inquiries does take place both within 
and across scenarios. Other performance measures do not seem to be 
affected. This suggests that information seeking style is governed 
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more by scenario content then by experience with CAT gleaned i^rom 
past scenarios. 

In general CAT appears to offer the most potential for learning how 
to attack particular families of CIFE’s. In addition it can be used 
to show pilot deficiencies in understanding the nature of GIFE's. 

F. GAT versus Paper and Pencil 

If one is careful to account for the differences in experience 
levels of the two groups of subjects, the results of the CAT experi- 
ments are generally consistent with those reported in the 1981 report 
on paper and pencil testing. Among the few differences noted are the 
following : 

1) The p/p group made fewer inquiries and explored fewer 
tracks in diagnosis than did the CAT group. This suggests 
that the computet form of testing may be less inhibiting 
and more user fciondly than face to face verbal exchange. 

2) The CAT group performed a wider search for destination 
information than did the P/P group. 

3) Although the most frequently selected airports were diffe- 
rent between the groups, the physical characteristics of 
the most popular airports were similar in both cases. 

G. PUVTO-tfAT HighXifthtS 

The PLATO-GAT experiments attempted to produce a GAT simulator 
expetiance using a computer terminal, in so far as resource management 
and information seeking were concerned. Both diagnosis and destination 
diversion scenarios were combined. This required the subject to diag- 
nose a probiem while entoute in a dynamic simulated environment in 
which alternate destinations could be selected. 

General observations about PLATO-GAT include the following: 

1) Pilots exhibit a wide variety of resource management 
styles similar to those found in the full mission simu- 
lation G/VT studies reported in 1981 (6) . 


2) Plloca show a strong preoccupation with problem symptoms 
and corrections which lead to poor positional awareness 

and poor choice of destination airports. This characteristic 
was also noted in the full mission GAT studies. 

3) Most pilots are reluctant to accept the fact that no dlag~ 
nosls or solution may be available in flight. 

4) Pilots do not appear to plan for problems in terms of 
constantly keeping enroute alternates in mind in case a 
problem does develop. Their management style is more 
reactive than pre-structured. 

H. Training Potential 

The computer aided testing instruments described in this report 
were developed as research tools to be used to better understand the 
decision making styles of pilots faced with critical in-flight events. 
However, from observations by the research team and from repeated 
comments by subject pilots, these tools may have even greater poten- 
tial for pilot training. 

CAT offers the possibility of self-paced instruction in acquiring 
basic aeronautical knowledge through an expanded version of the current 
knowledge survey. Such an expansion could easily include immediate 
feedback on the correctness of an answer, reasons for the correct re- 
sponse and references on availability of additional information. CAT 
also offers the opportunity to experience a variety of critical in- 
flight events from the safety of the computer terminal. In addition 
to a wealth of potential simulated decision experiences, CAT could 
be used to uncover pilot deficiencies in their current understanding 
of the nature of GiFE’s and to train pilots to develop more efficient 
information search habits. 
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